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PREFACE 


This report describes findings from a study to identify effective human teaching 
behaviors and ways of implementing them in intelligent tutoring systems. It is hoped that this 
type of research will both integrate the education research literature into intelligent tutoring 
systems development, and serve as a springboard for further research on effective teaching. 

The study was undertaken on behalf of the Armstrong Laboratory, Human Resources 
Directorate (AL^IRTT). Dr. Kurt Steuck was the Air Force task monitor. The authors would 
like to thank Dr. Steuck for initiating the investigation, for the use of his personal library, and 
for his invaluable guidance throughout the effort. 









SUMMARY 


The purpose of this study was to identify effective human teaching behaviors and ways 
of implementing them in an intelligent tutoring system GTS). This information was then used to 
develop a set of production rules to demonstrate how this instructional approach could be 
implemented in a computer-based format. It is hoped that this sample rule set can eventually be 
used to help develop an effective teaching template for use in a variety of intelhgent tutoring 
systems in the future. 

The study consisted of four major tasks. Table 1 describes these tasks and their 
respective inputs and outputs. 

Table 1. Summary of Project Tasks. 


TASK 

INPLJT(S) 

OUTPUTfS) 

1. Review of the literature on 
effective teaching 
behaviors (Chapter 2) 

Research literature on various aspects 
of effective teaching (e.g., presentation 
techniques, motivation, questioning, 
feedback) 

Identification of effective teaching 
behaviors 

2. Review of instructional 
approaches (Chapter 3) 

Research literature on selected 
instructional approaches (direct 
instruction, discovery learning, mastery 
learning, cognitive apprenticeship) 

Decision to use cognitive 
apprenticeship approach as a 
framework for incorporating effective 
teaching behaviors 

3. Review of forroats for 
knowledge representation 
on the computer (Chapter 

4) 

Literature on techniques for knowledge 
representation (e.g., production rules, 
semantic nets, predicate logic) and 
organization (e.g., blackboard, list of 
rules) 

Decision to use a list of production 
rules as a format for representing 
effective teaching knowledge 

4. Demonstrate how effective 
teaching knowledge can 
be implemented in a 
computer-based 
knowledge representation 
format (Chapter 5 and 
Appendixes A. B, and C) 

Outputs from Tasks 1-3 

1) An outline of instructional 
decisions involved in implementing 
the cognitive apprenticeship approach; 

2) a set of instructional variables; and 

3) rules which demonstrate how the 
cognitive apprenticeship approach 
could be implemented on a computer 


This study found that much of the existing educational research literature is applicable to 
intelligent tutoring systems development. However, translating effective teaching research into 
specific rules that can be executed by a computer was not easy, for a number of reasons. The 
major ones were: 1) there is so much information about effective teaching that it was not 
possible to review all the relevant areas; 2) the literature is often unclear as to exactly how to 
implement teaching behaviors; 3) the literature is particularly unclear on the issue of how to 
individualize instruction (which is usually a major reason for using a tutor in the first place); and 
4) computers are more limited than human teachers in their ability to handle multiple goals 
simultaneou.sly; and to continuously monitor the teaching situation. 
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Recommended areas for future research include: 1) operationalize and test the sample 
rule set on an intelligent tutoring system; 2) review other relevant areas of the literature (e.g., 
screen design, media selection, effective use of visuals); 3) m(»e research on how to 
individualize instruction (e.g., studies which examine how various iq)titudes, traits, and 
treatments interact); and 4) continue to track technology developments which would make it 
possible to use more powerful and versatile methods of representing knowledge on the 
computer. 


IDENTIFICATION OF EFFECTIVE TEACHING BEHAVIORS 
FOR INTELLIGENT TUTORING SYSTEMS 


L INTRODUCTION 

Intelligent tutoring systems (TTSs) arc computer-based systems which-in theory— 
utilize artificial intelligence (AI) techniques to provide highly individualized instruction, 
much like that of a human tutor. Tlie attempt to utilize AI in computer-assisted 
instruction (CAI) has been going on for more than twenty years, dating back to 
Carbonell’s seminal article, AI in CAI: An Artificial Intelligence Appro'ich to Computer- 
Assisted Instruction (1970). However, programming a computer to do the things that 
human teachers do has proven to be a very difficult task. 

One of the major problems in intelligent tutoring system development has been 
identifying what effective human teachers do. At least four different approaches have 
been taken to the development of the tutoring component of ITSs; 

• Inspired programmer. Tutoring principles are derived primarily from the 
developers' instincts about what constitutes good tutoring. This often 
happens when tutoring is not the main focus of the effort. For example, 
ITS researchers are sometimes particularly concerned with some aspect of 
ITS programming, such as techniques for detecting student errors, or 
fitting an ITS over an existing expert system (Brown and Burton's 
BUGGY, 1978; Qancey's GUIDON, 1984). Although this approach can 
produce acceptable instruction, it is not optimal. 

• Theory driven. Tutoring principles are based on some type of theory. 
Usually this is either cognitive theory (e.g., Anderson, Boyle, and Reiser's 
Lisp and Geometry tutors, 1985) or theory derived from computer-assisted 
instruction (e.g., Tennyson and Park's MAIS, 1987). If the theory is well- 
established, this can be an effective approach. Another advantage is that it 
gives developers a basis for using particular teaching strategics. However, 
a disadvantage is that most cognitive or educational theories are not 
comprehensive enough to generate all possible effective teaching 
strategies. Furthermore, evolution of the theory necessitates significant 
changes in the teaching strategies used. 

• Human tutor modeling. This approach involves closely observing human 
tutors' actions and making inferences about their reasoning, followed by 
construction of a computer model which mimics their actions (Collins, 
Wamock, & Passaflume's WHY, 1975). This approach can offer a high 
degree of validity i/there is a great deal of similarity between the role and 
knowledge of the human tutor and the expected role and knowledge of the 
ns. Validity will be compromised if the human tutor is not experienced, 
or if the human tutor only provides help with homework assignments, but 
the ns is expected to teach the course. 
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Even if similarity is liigh, a potential disadvantage of the expcn tuuw 
modeling approach is that an ITS which merely mimics human expert 
tutors may fail to utilize fully the media strengths of the computer, such as 
its animation, gaming, and simulation capabilities. 

• Integrative. This approach involves deriving tutoring principles from the 
available research literature on effective teaching behavim^ (e.g., O'Neil, 
Slawson, & Baker, 1991). Effective teaching behaviors are here defined 
as anything a teacher does that helps a student to learn what the teacher 
intend the student to learn. These may include techniques for motivating 
the student, explaining a concept more clearly, or encouraging the student 
to learn independently. Even activities which are not directly related to 
presenting subject matter, such as reaching study skills, may be considered 
to be effective teaching behaviors if they enable students to learn more 
effectively. 

This approach offers a certain degree of convenience, in that all 
developers have to do is read studies and distill principles. They do not 
have to set up a study or develop a theory-although they do need to be 
discriminating about the studies they choose. In addition, it allows 
developers to pick and choose well-grounded instructional principles for 
implementation within the ITS. 

There are several problems associated with the integrative approach, 
however. The body of educational research is incomplete and often 
inconclusive. In addition, much of the existing research has been 
conducted in classroom or laboratory situations, and therefore some 
findings may not be as effective when transferred to a one-on-one 
computer-based instructional environment. Finally, developers must 
figure out effective ways of combining teaching behaviors whose 
effectiveness was likely established in isolation. For example, both 
advance organizers and visual organizers (such as concept maps) have 
been found to be effective in some situations. But if used together for 
instruction, should they be presented separately or combined (e.g., using a 
concept mapping exercise as an advance organizer)? 

Each approach has advantages and disadvantages. Both the type of approach 
selected and the quality of its implementation can Lifluence the instructional 
effectiveness of the ITS. This repon describes a study by Armstrong Laboratory 
(AL/HRTI) which used the integrative approach. Although this approach has some 
potential problems as discussed earlier, it was selected because; 1) the arguments against 
it do not appear to be very strong (Goodyear, 1991); and 2) this approach has never been 
attempted in a comprehensive manner. 

The goal of this study was to produce domain-independent research-based 
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guidance for the development of the tutoring component of ITSs. This guidance will 
ultimately be made available to intelligent tutoring systems researchers, developers, and 
programmers in some readily computer-digestible form. It is hoped that this allow 
rrs development time to be shortened and educational effectiveness to rise. 

The following major tasks were undertaken in suppon of this research effon: 

• Review and assessment of the literature on effective teaching behaviors 
for applicability to implementation within intelligent tutoring systems 

• Review of instructional approaches and selection of one (cogiutive 
apprenticeship) to be used as a framework for implementing effective 
teaching behaviors 

• Review of computer-based knowledge representation formats and 
selection of one (list of production rules) for representing effective 
teaching knowledge 

• Integration of all the above into 1) an outline of cognitive apprenticeship 
decision processes, 2) a list of instructional variables; and 3) a sample set 
of production rules. 

The objectives, activities, and produces associated with these steps will be 
described in greater detail in the following chapters. 


n, REVIEW OF THE LITER.\TURE 

Broadly speaking, effective teaching behaviors can be defined as anything a 
teacher does that helps a student to learn what the teacher intends the student to learn. 
These may include techniques for motivating the student, explaining a concept more 
clearly, or encouraging the student to learn independently. Even activities such as 
counseling may be considered to be effective teaching behaviors, if they enable students 
to learn more effectively. Consequently, there is a vast amount of literature that pertains 
to effective teaching behaviors. 

However, not all effective teaching behaviors are applicable to intelligent tutoring 
systems. For exai]:q>le, empathizing with students' struggles or counseling students on 
their personal problems is best left to human teachers (at least for the present). 
Conversely, there are some things a computer can do-such as provide animation, 
simulation, and arcade-type ganKS-which the average human teacher would be hard-put 
to provide. Since the purpose of this task was to identify effective teaching behaviors for 
ITS implementation, the criteria in Table. Arere used to prioritize topics in the literature 
for review. 




Table 2. Criteria for Prioritizing Topics for Literature Review. 


PRIORITY 

EASE OF 

IMPLEMENTATION 

USEFULNESS 

EXAMPLE TOPICS 


easy 

broad 

Providing feedback 

medium 

easy 

restricted 

Use of simulation 

low 

difficult 

broad 

Cooperative learning 

non-‘'elect 

difficult 

limited or 
questionable 

Personal counseling 


Over 300 journal articles and books in the following areas were reviewed: 


• Elements of instructional presentation - e.g., advance organizers, concept 
maps, modeling, explaining, coaching, scaffolding, simulation 

• Motivation - e.g., assessment of motivation, computers and motivation, 
tutoring and motivation, attribution, locus of control, self-efficacy 

• Areas fDr student assessment (both before and during instruction) - e.g., 
educational history, academic motivation, general motivation, cognitive 
style, learning style 

• Questioning techniques - e.g., question types, levels of questioning, 
sequencing 

• Feedback - e.g., timing, content, response certitude, interaction with 
student characteristics 

• Learning strategies - e.g., metacognition, comprehension monitoring, 
rehearsal, elaboration, organizational strategies 

The more relevant articles and book chapters were summarized and evaluated, 
and teaching rules were derived based on the conclusions whenever possible. The 
purpose of developing teaching rules was to express knowledge about effective teaching 
in the form of instructions. These instructions could then be easily converted later to a 
form for computers. In deriving the rules, the researchers tried to be as specific as 
possible about what to do and when to do it, because unlike humans, computers need to 
be told exactly what to do when. For example, the following three rules were derived 
from an article on feedback (cf., Schultz, 1992): 

• If response certitude is high and the response is correct, little feedback is 
required. 

• If response certitude is high and the response is incorrect, provide 
informational feedback. 
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If response certitude is low, provide informational feedback. 


• 

In practice, however, it was usually difficult to obtain a high level of specificity, 
because the literature itself was not very specific. Often mles took the following form: 
To increase intrinsic motivation, allow student to make choices. This type of rule is less 
helpful, because it does not give much specific information about '^hen to allow choice. 
After all, it is always desirable to increase intrinsic motivation. 

The following sections summarize the literature reviewed, and the implications 
for intelligent mtoiing systems development These sections include: 

• Elements of instructional presentation 

• Motivation 

• Areas for assessment 

• Questioning strategies 

• Feedback 

• Learning strategies 


Elements Of Instructional Presentation 

Elements of presentation are like instructional tools. They are designed to help 
teachers and students reach speciEc learning objectives. This section addresses some of 
the more commonly-used elements of presentation. For each element, there will be a 
definition, a statement of purpose, a summary and evaluation of relevant research, and 
recommendations regarding its applicability to intelligent tutoring systems. The 
following elements will be discussed: 

• Advance organizers 

• Visual organizers 

• Mnemonics 

• Learner control 

• Scaffolding and fading 

• Modeling and telling 

Advance Organizers 


Definition 

An advance organizer is a presentational technique that attempts to encapsulate 
what a learner already knows in light of upcoming instruction. Supposedly, this 
technique facilitates learning and retention by providing the learner with explicit 
connections between existing knowledge and novel material (Ausubel, 1977). Generally, 
advance organizers are text-based passages, although graphic and aural organizers are 
also common (Luiten, Ames, & Ackerson, 1980). Advance organizers, according to 
Ausubel, have the following characteristics: 


i 
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They are presented prior to instruction. 

They focus on the ties between already learned and novel material. 
They are abstract, general, and inclusive statements of fact. 


Assimilation theory is often invoked to describe the effects of advance organizers 
(Ausubel, 1977). Theoretically, an advance organizer prompts the learner to transfer key 
information from long-term to short-term memory. Once in short-term memory, this 
information is activated. Activated knowledge anchors new knowledge by subsuming it 
under broad, pre-existing categories. In this way, cognitive structure is fleshed out. We 
can see that abstract, general, and inclusive statements are better anchors, according to 
assimilation theory, because they prompt the learner to draw upon broader and therefore 
more subsuming sources of knowledge. The organizer acts as a bridge between what the 
leame, already knows and the instructional content. Advance organizers, then, tailor 
new knowledge to existing cognitive structure (Ausubel, 1977; Krahan & Blancher, 
1986). 

Purpose 

Ausubel (1977) has most succinctly defined the purpose of the advance organizer. 
According to him, 

...the principal function of the organizer is to bridge the gap between what 
the learner already knows and what he needs to know b^ore he can 
successfully learn the task at hand. 

In shon, advance organizers are employed to facilitate meaningful learning and promote 
retention of novel material by causing an active integration of new and old information 
(Krahan & Blancher, 1986). 

Summary and Evaluation of the Research 

Generally, advance organizer research focuses on: 1) learner characteristics, 2) 
conditions of learning, and 3) organizer characteristics. 

Learner Characteristics 

It is likely that certain learner characteristics (e.g., age) moderate the 
effectiveness of advance organizers. Possibly, advance organizers work for some types of 
learners but not others. If this is indeed the case, then the design and use of advance 
organizers must be carefully considered. The effects of grade level, subject ability, and 
subject knowledge are examined. Recommendations are presented at the end of the 
section. 

Grade level. Stone (1983), using meta-analysis, found mean effect sizes for 
preschool (1.01), elementary (.64), junior high (1.39), high school (.45), and college 
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students (.49). Luiten, Ames, & Ackerson (1980) found smaller effect sizes favoring the 
use of advance organizers for learning and retention criteria across grade level. Mean 
effect sizes ranged from .17-.33 indicating that the average individual receiving an 
advance organizer performed better than approximately 57-63% of control subjects. 
These results suggest two things. First, advance organizers seem to improve 
perfonnance across grade level. Second, younger populations tend to benefit most from 
advance organizers. The second fmding is problematic, however. Large standard errors 
were associated with these effect sizes. This means that if we were to construct 
confidence intervals around these estimates of effect size, the intervals would be quite 
large and they would overlap. Therefore, it is difficult to say with any conviction that 
grade level moderates the effectiveness of advance organizers. We can reasonably 
conclude that advance organizers facilitate learning across grade level. Despite the 
obvious variability in the reported effect sizes across grade, certain methodological 
factors (e.g., number of studies, sizes of the standard errors) make conclusions about 
specific grades risky. 

Subject ability. Luiten et al. (1980) examined the effectiveness of advance 
organizers for three subject ability groups: high (.23), medium (.08), and low (.13). In 
similar work. Stone found mean effect sizes for high (.34), medium (.64), and low (.26) 
ability students. Again, standard errors in both smdies were large enough to prohibit 
conclusive interpretations. Th? only conclusion that we can reach is that, generally, 
advance organizers promote learning and retention across ability levels. 

Subject knowledge. Subject knowledge is similar to ability except that it is 
domain-specific. Stone repons mean effect sizes for high (.24), medium (.27), and low 
(.032) knowledge students. Simply looking at these figures suggests that advance 
organizers are more effective for non-novice students although the number of studies in 
each category averages only 5. Interpretation of results based on a study sample size of 5 
is problematic. Unfortunately, it is difficult to say anything about the relationship 
between subject knowledge and the effectiveness of advance organizers. 

Conditions ofLeamine 

Conditions of learning refer to characteristics of the learning environment which 
are defined by the curriculum or teacher. 

Subject area. Luiten et al. (1980) examined the effectiveness of advance 
organizers for learning and retention criteria in mathematics, and the physical, biological, 
and social sciences. The results are shown in Table 3. Once again, study sample size 
and error confound interpretation. For example, a 95% confidence interval for retention 
criteria in mathematics would provide the following interval: (-.048 < ES < .248). The 
lower limit is negative. With an observed effect size of .17, it is possible that advance 
organizers may actually hinder retention in mathematics. However, it is reasonable to 
conclude that in general, advance organizers slightly facilitate both learning and retention 
across subject areas. 
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Table 3. Mean Effect Sizes of Advance Organizers by Domain and Criterion. 


1 CRTTERIAFOR 
ASSESSING 
EFFECTIVENESS 

MATHEMATICS 

PHYSICAL 

SCIENCES 

BIOLOGICAL 

SCIENCES 

SOCIAL 

SCIENCES 

LEARNING 

.10 

.15 

.11 

.13 

RETENTION 

.17 

.50 

.18 

.26 


Organizer C iracteristics 

Characteristics of the organizers themselves may moderate effectiveness. 
Characteristics examined include: mode of presentation, type, style, hierarchy type, 
source, and operational level. 

Mode of presentation. Luiten et al. (1980) compared written and aural advance 
organizers in terms of effectiveness. Both had a facilitative effect on learning. Written 
advance organizers had a facilitative effect of .17, while aural advance organizers had a 
facilitative effect of .37. This effect size difference of .20 translates to a 7.7% 
improvement in average performance over control when aural organizers are used in 
place of written organizers. It is not always possible or even desirable to use aural 
organizers, however. 

Type. Kloster & Winne (1989) compared the effects of different types of advance 
organizers on learning from text. Concept, analogy, outline, and dummy organizers were 
employed. They found no main effects by organizer type although the effectiveness of an 
organizer was related to the student’s ability to relate new information to the organizer. 
They concluded that the true advance organizers like concept and analogy organizers 
resulted in superior performance only when students used them efficiently. Often, 
however, the students had a hard time connecting the information from the lesson to the 
organizer. This suggests that Ausubel's claim that organizers should be general is not 
necessarily true. 

Style. Expository and comparative advance organizers seem to promote learning 
better than mixed or narrative organizers. Stone found effect sizes of .80 (expositoiy), 
.88 (comparative), .18 (mixed), and .29 (naixative). These results suggest two things. 
First, organizers should adhere to a single style. Second, explanatory and comparative 
organizen are superior to narrative ones. 

Hierarchy type. Hierarchy type refers to whether or not the advance organizer 
subsumes the instructional content (Stone, 1983). That is, does the organizer provide the 
learner with information which is more abstract and general than the instructional 
content? Assimilation theory would predict that subsuming information would greatly 
enhance the effectiveness of the organizer. Stone, on the other hand, found that non¬ 
subsuming organizers were slightly better. 

Source. Organizer content can be derived from several sources including 
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previously covered material or future instructional content. Source refers to whether or 
not the content of the organizer was drawn directly from the instructional material or 
whether it was designed to bridge old and new knowledge. Stone's results suggest that 
organizer^ drawn direcdy from the instrucdonal content arc slightly more effective 
(Stone, 1983). Again, this finding contradicts assimilation theory. According to the 
theory, information which bridges old and new knowledge should be more effective. 
This result suggests that organizers should be drawn directly from the instructional 
material. 

Operational level. Operational level refers to whether the organizer is concrete or 
abstract. Ausubel claims that abstract organizers are more effective because they activate 
subsuming concepts. Stone's results contradict Ausubel's position (Stone, 1983). She 
found that concrete organizers facilitated learning more than abstract ones. This finding 
suggests that organizers should focus on the specific and concrete features of the 
knowledge to be learned. 

All in all, advance organizers seem to facilitate learning and retention across a 
variety of circumstances. Unfortunately, the methodological characteristics of many 
advance organizer studies prohibit full interpretation of the data. Nonetheless, the 
following general conclusions emerge: 

• Advance organizers facilitate learning and retention for most learners. 

• Advance organizers facilitate learning and retention across most 
disciplines. 

• Concrete and non-subsuming organizers seem to promote learning better 
than abstract and subsuming organizers. 

• Expository and comparative organizers seem to be better than narrative 
organizers. 

• Ti.e use of true organizers (i.e., concept and analogy organizers) should be 
reserved for use with higher ability students. 

• Oftentimes, it desirable to use lesson content as the source for organizer 
material. 

Applicability to Intelligent Tutoring Systems 

Advance organizers, including graphic and aural forms, are applicable to 
intelligent tutoring systems. Unfortunately, the resolution of the research is poor; 
therefore, we cannot individualize organizers to fit many learner characteristics. 
Additional research and cumulative studies are necessary to clarify the relationships 
between organizers, the learner, and achievement. 
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Concept Maps and Other Visual Organizers 


Dilutions 

Supposedly, visual organizers are graphic representations of innate thought 
processes (Qarke, 1991). As such, they are metacognitive tools; they help students to 
recognize and control intellectual processes which bring meaning to lesson content aiul 
promote higher-order thinking skills. There are two general classes of visual (Hganizers: 
bottom-up and top-down. Bottom-up organizers are inducdvely-based That is, diey help 
students to organize informadon so that they can draw inferences about trends, related 
concepts, group characterisdcs, and theories. Oarke (1991) identifies the following 
examples of bottom-up visuals: 

• Time lines: Events are plotted along a continuum in chronological order. 
Time lines allow students to observe trends and identify cause-and-effect 
reladonships. 

• Web ^grants: A main idea is placed in the hub or center of the web. 
Related concepts and facts are placed around the perimeter of the hub. For 
example, the hub of Figure 1 might represent the domain mathematics 
while the perimeter sites might represent algebra, geometry, statistics and 
so forth. Each of the perimeter sites could be further sub-divided. Web 
diagrams allow students to examine the reladonships between ideas, 
concepts, and facts. 



Figure 1. Example of a Weu £)!...^.'am 
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Grids, pie charts, and graphs: The frequency of some event is recorded in 
graphical fomL Grids, pie charts, and graphs allow students to see trends 
in quantitative data. 

Venn diagrams: A set of overbq)ping circles is drawn as shown in Figure 
2. Each circle represents some entity. Each endty has some uniqueness 
(e.g.. A). The areas of overlap (e.g.. ab, abc) represent shared or common 
characterisdcs. In Figure 2. circles A. B, and C might represent 
mathematics, physics, and astronomy. Venn diagrams allow students to 
sort informadon into muldple categories and compare various enddes in 
terms of relevant characterisdcs. 



Figure 2. An Example of a Venn Diagram 





• Inductive towers: Sets of facts are structured hierarchically under a 
theory. In Figure 3, the circles might represent a collection of 
observations about motion which successively build up to Newton's Laws. 
Inductive towers allow students to connect factual statements, build 
theories, and test predictions (see Figure 3). 



Figure 3. An Example of an Inductive Tower 


Top-down organizers are deductively-based. They help students apply rules, test 
hypotheses, and make decisions. Clarke (1991) identifies the following top-down 
visuals: 

• Weighing scales, continuum lines, and prolcon charts: Arguments for and 
against a theoiy, concept, or idea are charted and compared. These 
methods allow students to systematically examine the pluses and minuses 
associated with an argument. 

• Force field diagrams: Antithetical forces are charted opposite one another. 
Force field diagrams allow students to see which side of an issue seems to 
have the most support. 
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* Causal chains: Sets of causes are linked to an observed outconie. Causal 
chains allow students to nxxiel processes and phenomena. 

* Decision trees and IF-THEN flow chans: Sequential processes are 
charted. Decision trees and IF-THEN flow charts are like road maps. 
They allow students to follow a simple sequence to reach some outcome. 

* Concept maps: Related ideas, concepts, and facts are arranged 
hierarchically. Connections between nodes are made explicit Concept 
mtqjs allow students to visualize connections between related concepts. 

Concept maps have received the most attention in the literature by far. Therefore, 
this remainder of this section will focus on them. 

The concept map is a specialized visual organizer designed to mimic hierarchical 
memory structures. Generally, concept maps consist of nodes and links. Nodes are 
concepts and links are relations. Nodes are arranged in order of inclusiveness. That is, 
the most general concepts are placed at the top of the hierarchy while more specific 
concepts or examples are placed at progressively lower levels of the hierarchy. Nodes 
are connected by links which indicate the nature of the relationship between concepts or 
examples (Novak, 1985). For example, a link between higher and lower-level nodes may 
have the descriptor, characteristic of. This means that the lower-level node is a 
characteristic of the higher-level concept. 

The impetus for concept mapping, like advance organizers, comes from Ausubel's 
theory of meaningful learning. Ausubel asserts that meaningful learning occurs when 
new knowledge is accurately related to and subsumed by existing cognitive structure in a 
non-arbitraiy, non-verbatim fashion (Ausubel, 1977). Supposedly, concept mapping 
accomplishes this goal by helping students to relate abstract, inclusive concepts with 
specific instances (Qarke, 1991). In a sense, the concept map is microcosmic 
representation of Ausubel's view of human cognition. In concept mapping, general and 
inclusive concepts subsume specific ones. That is, general and inclusive concepts are 
progressively differentiated or divided into smaller and smaller units. As this happens, 
the concept hierarchy grows, branching both vertically and horizontally. The same is 
true of human memory. Gradually, experience differentiates broad categories of 
knowledge. General, inclusive, and subsuming concepts are pushed to the top of the 
hierarchy while specific examples are pushed towards the bottom (Ausubel, 1977^ 

Purpose 

The ultimate purpose of using concept maps and other visual organizers is to 
facilitate meaningful learning and promote metacognitive development. Supposedly, 
visual organizers accomplish these goals by mimicking innate thought processes, 
focusing on higher-order thinking skills, and taking advantage of the unique human 
capacity for visual learning (Clibum, 1990). Concept maps can also be used as 
evaluation tools. When concept maps are used to evaluate domain knowledge, it is often 
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easy to detect where student misconcepdons lie (Moreira, 1985; Zcitz & Anderson- 
Inman, 1992). 

Summary and Evaluation of Findings 

Visual organizers (e.g., concept maps) seem to support student comprehension 
and encourage the development of higher-order thinking skills (Clarke, 1991). Not 
surprisingly, Novak, Gowin, & Johansen (1983) found that higher ability students tended 
to be the most effective mappers. They also found that the use of mapping strategies 
facilitated novel task transfer over traditional teaming conditions (Novak et al., 1983). 
Hienze-Fry and Novak (1990) did not find significant differences between mapping and 
non-mapping groups on measures of learning and retention. However, error analysis 
revealed that mapping helped to clarify learning by reducing the numbers of errors made 
by mappers (Heinze-Fry & Novak, 1990). Similarly, Lambiotte, Peel, and Dansereau 
(1992) found that concept mapping did not improve performance on a simple test of 
recall, but that it did improve higher-thinking skills. 

Other researchers, such as Pankratius (1990), have found that concept mapping 
does result in superior achievement over non-mapping strategies. One possible 
explanation for this discrepancy lies in the fact that certain learner characteristics seem to 
mediate the relationship between concept-mapping and post-test achievement. For 
example, Stensvold and Wilson (1990) found that low verbd ability ninth grade students 
performed significantly better under concept mapping strategies than non-concept 
mapping students. Interestingly, this effect did not hold up for high verbal ability 
students (Stensvold & Wilson, 1990). The literature seems to suggest that requiring 
students to concept map does not, in and of itself, improve performance on simple tests 
of learning. It does, however, appear that mapping strategies facilitate higher-order 
thinking skills and promote long-term retention and transfer. Studies examining the 
relationships between concept mapping, learner characteristics, and achievement must be 
conducted. 

Applicability to Intelligent Tutoring Systems 

Visual organizers are well-suited for use in intelligent tutoring. The computer 
provides a unique medium for instantiating visual organizers. First, the logical and 
hierarchical structiu-e of the concept map lends itself to computer-based diagnosis. 
Second, the computer can graphically represent several attempts at mapping some set of 
knowledge for a single student much like Collins' abstracted reification. Presumably, this 
form of reflection would facilitate metacognitive development. Third, the computer can 
easily model proper mapping skills. 
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Mnemonics 


Definition 

Mnemonics are memory devices which help the learner to encode and recall novel 
information (Davis, 1983). There are three general mnemonic strategies or ways to 
facilitate the encoding process: 

• grouping strategies 

• imagery strategies 

• rhyming strategies 

Information can be grouped or categorized according to some scheme. For 
example, if we have a list of animals to remember, we can group them into meaningful 
categories like carnivores, herbivores, and omnivores. Then, we can subdivide each 
category with specific examples. CSrouping the animals facilitates the memory process 
by establishing meaningful ties between broad and speciHc forms of knowledge. This 
type of mnemonic utilizes a grouping strategy. Another grouping strategy involves 
arranging the first letter of the items in a list to form an imaginaiy word. For example, 
introductory algebra teachers often teach the FOIL method for simplifying certain 
algebraic expressions. Each of the letters of the mnemonic specifies an operation that the 
student is to perform. This form of grouping is probably the most common mnemonic 
strategy. 

The second type of mnemonic coding strategy involves the use of mental 
imagery. Imagery strategies require the learner to associate the concepts to be learned 
with interesting images. The one-bun, two-shoe, three-tree strategy utilizes a common 
set of associations. The learner must initially learn the association scheme (i.e, one-bun, 
two-shoe etc.). Then, he or she may use the scheme to anchor novel information by 
forming visual associations between the information and the nouns listed in the scheme. 
In order to remember our list of animals, we would form connections between the first 
animal on OUT li<:t and bun. Suppose the fast animal was a lion. We could remember 
lion, the king of beasts, by imagiiung it wearing a bun as a crown. Next, we would pair 
the second item on the list with shoe, and so on. Another type of imagery mnemonic is 
called the method of loci. The method of loci requires that the learner associate concepts 
with locations. For example, we might remember our list of animals by associating each 
one with q)ecific landmar ks along a road that we commonly travel. The lion could be at 
the comer grocery store, the owl at the city park and so forth. To recall the list of 
animals, we would simply take an imaginary trip along our pre-memorized route. 

The final type of mnemonic device is rhyming. Rhyming requires that either the 
teacher or student devise short rhymes which capture the target knowledge. Columbus 
sailed the ocean blue, in fourteen-hiAulred and ninety-two is a well-known example of 
this type of mnemonic. 

Mnemonics, in short, are memory tools. They are designed to help us encode and 
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recall large quantities of information by tying that information to pie-memorized 
anchors. 


Purpose 

Mnemonics are devices which serve two purposes. First, they facilitate the 
transfer of knowledge from short-term to long-term memory. Second, they facilitate the 
recall of information from long-term memory (Davis, 1983). 

Summary and Evaluation of Findings 

Levin, Morrison, McGivem, Mastropieri, & Scruggs (1986) found that the use of 
mnemonic strategies resulted in superior post-test recall of science material for eighth 
grade students. Not only did subjects in the mnemonic condition outpeiform their 
counterparts, they were also aware of the benefits associated with the use of mnemonics. 
This suggests that they may possibly use them in the future. Another study indicated that 
the method of loci, or more specifically, the spatial-arrangement mnemonic also resulted 
in superior recall relative to other methods (Bellezza, 1983). Words were presented in 
same-arrangement and different-arrangement conditions. Different-arrangement 
conditions resulted in distinctively arranged sets of words embedded in maps. Bellezza 
found that distinctive word arrangements facilitated both recall and retention in coUege- 
age students. Therefore, it seems that emphasizing the uniqueness of a set of concepts by 
tying them to unique conditions results in superior recall. 

Applicability to Intelligent Tutoring Systems 

Computers would be an especially effective means of presenting graphical, 
imagery-based mnemonics to students. Static graphic images and animation could be 
used to promote mental imagery. 


Learner Control 


Definition 

The term learner control is essentially self-explanatory; sometimes it is desirable 
to relinquish control of certain features of the instructional process to the student. 
Generally, the learner can exert control over four features of instruction; 1) lesson pace, 
2) lesson content, and 3 & 4) the number and sequence of instructional events. Allowing 
learner control of instruction rests on the supposition that learners make good choices 
(Carrier, 1984). Unfortunately, they often do not. Therefore, before instituting a system 
of learner control, we must carefully consider the situation. The literature offers some 
findings which clarify this process. 
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Purpose 

Learner control is often mentioned in conjunction with individualized instruction. 
Simply, if mdividuals are allowed to determine key features of the instructional process, 
then it is assumed that they will effectively tailor it to suit their needs. The promise of 
learner control is twofold. First, learner control should hypothetically increase 
achievement because students are allowed to tailor instruction to their needs. Second, 
learner control should contribute to positive student motivation and attitudes because 
students have the perception of control. 

Summary and Evaluation of Findings 

First, we will discuss the effects of learner and program control on student 
achievement Program control refers to the more traditional approach of computer-based 
instruction. Specifically, the software, and not the learner, determines the characteristics 
of instruction. Kenzie, Sullivan, and Berdel (1988) found that learner control of content 
review in a computer based science lesson resulted in superior post-test achievement 
relative to program control. Other studies, however, have found contradictory results. 
For example. Carrier (1984) notes that complete learner control generally results in lower 
post-test achievement scores because learners select methods which they think will 
require less work, concentration, or time. Therefore, they opt for inferior methods of 
instruction and sacrifice achievement. Sometimes, given the option, they even opt 
themselves out of instruction all together. 

The findings regarding the effects of learner control on achievement arc mixed. 
One reason for this uncertainty is that a variety of learner characteristics seem to mediate 
the relationship between learner control and achievement. For example, older and more 
experienced learners seem to be better suited to learner control conditions than novice 
learners (Carrier, 1984). In a similar vein, Ross & Rakow (1981) found that students 
with low prior Knowledge performed better under program control. Other studies have 
indicated that when the instructional material is simple that learner control is superior, 
but when the material is complex that program control results In higher achievement 
(Steinberg, 1989). 

Research suggests that learner control of screen viewing does not affect 
achievement, but that learner control of the amount of material presented on the screen 
produced achievement gains (Steinberg, 1989). Lower ability students tended to choose 
higher density screens while average and higher ability students tended to choose lower 
density screens. Therefore, all students received the amount of information that they 
needed. High ability students did not have to sit through boring reviews of old material 
and low ability students had access to the additional information that they needed. 

"iDther learner characteristics are tied to the effectiveness of learner control. An 
individual's locus of control is one such variable. Holloway (1978) examined locus of 
control, learner control, and achievement. He found that individuals with an internal 
locus of control performed better under learner control conditions. This finding makes 
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sense; individuals who - guided from within are more likely to prosper when they are 
free to make instructional choices. 

Field independence is another psychological variable which app>ears to be related 
to the effectiveness of learner control (Carrier, 1984). Field independence is correlated 
with the cognitive restructuring of instructional stimuli. Field independents restructure a 
field of stimuli when it is beneficial to do so; they are less likely to accept a situation as 
given. They like to change situations to fit their needs. It seems logical, then, that field 
independents would prefer learner control because it allows them tailor certain features 
of the instructional stimuli to suit their needs. Field independents generally require more 
practice time than field dependents because they tend to use trial and error methods more 
often. Carrier, Davidson, Higson, and Williams (1977) allowed learners to choose 
expanded definitions, additional explanation, additional practice, or analytical feedb.rek. 
Choice of options did not improve achievement. Field independent learners did choose 
more options than field dependent learners, however. Therefore, if nothing else, learner 
control allowed learners to tailor instruction to suit their needs. 

Supposedly, learner control facilitates both student motivation and school 
attitudes. Ross and Rakow vl980), for example, found that learner control promoted 
positive attitudes towards learning. Steinberg (1989) summarized some of the key 
findings in learner control research. One point she makes is that, generally, learner 
control in computer-based instruction results in higher smdent motivation (Steinberg, 
1989). Increases in motivation, however, are not usually reflected in achievement gains. 
In fact, as we have stated, the use of learner control with certain populations can actually 
reduce achievement. Student motivation is a desirable outcome in its own right, but it is 
probably not wise to sacrifice achievement for motivation. 

Learner conuol is a complex issue which we have only briefly discussed here. 
However, we can offer the following guidelines regarding the use of learner control in 
intelligent tutoring systems: 

• Use learner control with novices, low-ability smdents, or in complex 
domains with caution. 

• Consider using learner control with advisement instead of pure learner 
control. 

• Focus learner control on benign aspects of instruction to provide the 
learner with the perception of control and minimize achievement losses 
associated with poor learner choices. 

Applicability to Intelligent Tutoring Systems 

Learner control is essentially a computer-based technique. Fhat is, learner 
control is primarily applied in computer-based instruction; it would be nearly impossible 
for a classroom teacher to allow each student to control important features of the 
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instructional process. Therefore, by definition, learner control is applicable to intelligent 
tutoring systems. The recommendations presented above should be followed, however. 

Scaffolding and Fading 


Definitions 

Scaffolding refers to a pnxess that gradually enables a learner to do something 
that he or she is not capable of doing alone (Palincsar, 1986)-like training wheels on a 
bicycle. Scaffolded instruedon is based on Vygotsky’s zone of proximal development. 
Vygotsky (1978) says the zone of proximal development is: 

the distance between the actual developmental level as determined by 

independent problem solving and the level of potential development as 

determined through problem solving under adult guidance, or in 

collaboration with more capable peer 

Vygotsky's definition of zone of proximal development reflects his two pan theory of 
learning. SpeciHcally, Vygotsky believes that learning has social and developmental 
components. Learning is a developmental process in that learners must navigate the zone 
of proximal development One side of the zone represents the current state of knowledge 
while the other side represents a more mature, developed state of knowledge. Learning is 
a social process in that it requires interactions with those who hold the knowledge. Thus, 
scaffolded instruction is based upon a social and developmental theory of learning. 
Palincsar (1986) identifies the following attributes of the process of providing scaffolded 
instruction: 

• It begins with the selection of a learning task that is emerging in the 
learner's repertoire but not yet mature. 

• The task is evaluated to determine its difficulty. 

• The teacher determines how to make the task easier, notes what parts of 
the task to accentuate in instruction, and determines how to sequence 
piesentation. 

• During instruction, the teacher employs modeling, questioning, and 
explanation to clarify the learning task. 

• Instruction and learning occur in successive approximations. 

• Emphasis is on student participation/interactive leaming/collaboration. 

• The purpose of evaluation in scaffolded instruction is to provide the 
teacher with an estimate of student knowledge and progress. 
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Support is provided based on task difficulty and student knowledge. 
It is accompanied by fading. 


• 

Scaffolded instruction requires fading. Fading is accomplished by gradually 
withdrawing the metaphorical scaffold as the learner demonstrates increasing 
competence (Palincsar, 1986). Without fading, scaffolded instruction would be largely 
ineffective because some students would merely wait for the expert to solve the problem 
for them. The fading of support requires students to take an increasing amount of 
responsibility for learning and problem solving. 

Purpose 

The main purpose of scaffolded instruction is to facilitate the generalization of 
knowledge to less structured contexts (Applebee & Langer, 1983; Palincsar, 1986). 
Implicitly, scaffolding is also supposed to promote learning and retention through an 
interactive and nurturing learning environment. 

Summary and Evaluation of Findings 

Scaffolded instruction has proven successful in several empirical studies, 
although primarily with younger, lower ability populations. Palincsar and Brown (1984), 
for example, repon that reciprocal teaching facilitates learning, retention, and transfer in 
seventh grade students classified as poor comprehenders. Palincsar (1986) reported 
similar findings. It is important to note, however, that these smdies examined the 
effectivei-ess of reciprocal teaching as an instructional approach and not scaffolding 
itself. While the scaffolding/fading paradigm is a large part of reciprocal teaching, other 
characteristics of the approach may have confounded the results. Lajoie and Lesgold 
(1989) tested the effectiveness of an apprenticeship tutor on F*15 technical trainees' 
troubleshooting skills. They found that subjects who spent at least twenty hours working 
on SHERLOCK were as proficient in troubleshooting the test station as technicians with 
four years of experience (Lajoie & Lesgold, 1989). Again these findings do not 
represent the effects of scaffolding alone. Therefore, it is difficult to conclude with 
complete certainty that scaffolding is an effective tool although the preponderance of the 
evidence suggests that it is. 

Applicability to Intelligent Tutoring Systems 

Scaffolding is a useful instructional tool which can be instantiated in ITSs. The 
successful implementation of scaffolding in ITSs hinges on our ability to build complex 
and dynamic student models. The computer must be able to diagnose and respond to a 
variety of student behaviors. 
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Telling and Modeling 


Dilutions 

There are three general ways in which teachers impart knowledge to students. 
First, and most simply, they hold students responsible for learning by requiring them to 
do something. Second, they tell students important information. Third, they show them 
skills or processes associated with some knowledge. This section discusses the 
characterisdcs of telling and showing as elements of instructional presentation. 

Telling is often an efficient and effective means of transmitting information, 
although it can facilitate idiosyncratic learning, rote memorization, and diminish the 
importance of multiple perspectives (Schank & Jona, 1991). Telling requires the teacher 
to identify important information and pass it along to students. In this sense, the teacher 
plays an active role in the process while the student does not Since the teacher identifies 
what information is important, telling is efficient; the student does not have to spend a lot 
of time sorting through irrelevant material. Unfortunately, telling is effective only when 
students actually understand what they are told. Often ^ey do not. Therefore, telling 
content does not ensure learning (Winne, 1989). In fact, it can facilitate idiosyncratic 
and superficial leanung of content because students only interact with the knowledge 
passively. They do not question it, they merely accept it Telling is an appropriate 
instructional strategy under the following conditions: 

• when only a small amount of information is presented 

• when the lesson is ongoing 

• when student requests information 

Showing or modeling is often a more useful approach than telling. Five 
characteristics of modeling make it superior to telling done. These characteristics 
promote meartingful leanung, retention, and transfer (Anderson & Armbruster, 1990; 
Brophy, 1986; ShueU, 1988; Walberg, 1991): 

• modeling requires both showing and telling of knowledge 

• modeling is authentic 

• modeling promotes multiple perspectives 

• modeling promotes active student participation 

• modeling affords students a wealth of instructional cues 

Modeling requires an instructor to simultaneously demonstrate a skill or process 
and explain its purpose (Collins, 1988). Modeling, then, utilizes both the showing and 
telling of knowledge which provides the learner with more complete information than 
telling alone. Additionally, modeling takes advantage of our unique capacity as visual 
learners. Modeling is superior to telling, in this sense, because it includes the showing 
and telling of information. Telling, on the other hand, does not require showing. 

Authenticity refers to realism in instruction. Instruction is traditionally situated in 
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the context of academia. Learners are led to believe that school knowledge does not have 
applications outside of the classroom. Modeling, on the other hand, situates knowledge 
in the context of its use. The instructor does not merely provide information, he or she 
demonstrates the application of that knowledge. The smdent receives explicit instruction 
in the real-world uses of knowledge. Therefore, superficial learning is minimized. 

The use of multiple perspectives in instruction refers to the examination of more 
than one viewpoint for a given topic. Multiple perspectives in instruction increases the 
meaningfulness of learning by increasing the diversity of material covered. For example, 
an instructor might model alternative approaches to solving the same algebra problem. 
The student then sees two ways to solve the same problem. Even if the student fails to 
remember both strategies, the exercise has been useful because he or she knows that 
alternatives exist. 

Modeling encourages more active learning than telling does, for two reasons. 
First, skills are demonstrated because students will be required to perform them. This 
encourages active participation because students usually want to perform well. Second, 
students often determine what will be modeled by asking questions and offering differing 
perspectives. Therefore, they have an active role in deciding what will be taught. 

Finally, modeling offers students a plethora of instructional cues. Cues are timely 
prompts provided by the instructor which are designed to help the student overcome 
some problem. Walberg (1991) found an effect size of 1.25 for instructional 
interventions utilizing cues. In other words, students who received instructional cues did 
better than approximately 89% of those who did not receive cues. Not only does 
modeling provide the learner with obvious cues about how to do things; it also provides 
subtle cues about practical aspects of performance that an instructor might overlook if he 
or she were telling content. 

Modeling is a flexible and comprehensive instructional method. It provides 
learners with a rich source of information which is simultaneously authentic and diverse. 
Generally, three benefits are associated with the use of modeling: 1) students see expert 
solutions to problems, 2) students can integrate what happens with why, and 3) invisible 
parts of the process are made visible (Collins, 1988). 

Purpose 

Telling and modeling are methods of instnictional presentation. As such, they are 
designed to facilitate meaningful learning, retention, and transfer. 

Summary and Evaluation of Findings 

The effectiveness of modeling and telling are generally not tested independently. 
Rather, they are tested as components of larger models of instruction like cognitive 
apprenticeship and direct instruction. As mentioned previously, both of these larger 
models have been studied empirically and both seem to be effective instructional models. 
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Modeling, like cognitive apprenticeship, is a richer, more flexible, and more 
comprehensive approach. 

Applicability to Intelligent Tutoring Systems 

Both modeling and telling are applicable to intelligent tutoring systems. In fact, it 
is nearly imperative to use both of them because each is useful under different 
circumstances. Modeling should be used: 

• at the beginning of a new lesson to structure subsequent instruction 

• when the knowledge to be taught is general 

• when the learner needs to be reminded of the big picture 

• when the learner requests it 

• early in instruction when fading has not begun 

Telling should be used; 

• during an ongoing lesson 

• when the knowledge to be taught is specific 

• when the learner requests it 

• when fading of assistance is a system goal 

Motivation 

Like the proverbial horse, you can lead a student to class, but you can’t make him 
learn. The purpose of investigating the topic of motivation was to identify what factors 
affect a student's willingness to leam, and what can be done-in the context of an 
intelligent tutoring system-to sustain and/or increase a student’s level of motivation. 

Definitions 

Motivation can be defined as that which gives direction and iraensity to behavior 
(E. Gagne, 1985). A basic assumption of motivational theories is that all behavior is 
motivated. 

Motivation can be intrinsic or extrinsic. It is intrinsic if the goal or reward is the 
task itself. For example, in education, learning for the sake of learning is an instance of 
intrinsic motivation. Motivation is said to be extrinsic if external rewards or punishment 
must be used to influence behavior, because in this case, the goal is external to the task. 
Examples of extrinsic motivators include grades, prizes, smiles, and words of 
encouragement In general, most theorists believe that intrinsically motivated learning is 
more desirable than extrinsically motivated learning (Davis, 1983). In practice, however, 
any type of motivation is better than none. 

Motivation can be either a personality trait or a temporary state (Davis, 1983). 
As a trait, it is individualized, internalized, and closely related to attitudes regarding the 
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value of learning, aspirations, educational goals and self-esteem. Such individual 
differences are stable and lasting, and may influence not only schooling, but also many 
other spheres of a person's life. As a state, motivation is temporary and situational, 
depending on stimulating or dull events in the environment Clearly it is these states of 
motivation over which the teacher has the greatest control. 

Summary and Evaluation of Research 

The research for this study concentrated on two areas: 1) factors affecting 
motivation, and 2) tools for assessment of motivation. 

Factors Affecting Motivation 

There are a number of factors which have been shown to influence motivation. 
These include: satisfaction of basic needs, curiosity, cognitive dissonance, desire for 
competence, need for achievement, attributions, locus of control, self-efficacy, nature of 
the goal, observational learning, orientation toward learning, gender, domain, training, 
and attitudes about computers. 

Satisfaction of basic needs. Because all of the factors affecting motivation are 
related to the desire to satisfy some type of innate need, this term is a misnomer. 
However, it is used here to suggest that some needs are more basic than others, and that 
motivation is related to the level of urgency of the need that must be met According to 
Maslow's (1970) hierarchical need model, human motivation is based upon level of need, 
beginning with basic physiological and security needs, progressing through the need for 
belonging, esteem, and knowledge, and culminating with the need for self-actualizrtion. 
Maslow states that when a higher level need conflicts with a lower level need, the lower 
need takes precedence. Within the realm of education and learning, this simply means 
that a student's lower needs must be met before academic needs will appear. On the 
other hand, if the higher order needs are not met in the educational setting, the student 
will either barely tolerate the learning process or quit. 

Curiosity. Curiosity is a source of motivation, because it leads to exploratory 
behavior aimed at reducing uncertainty (E. Gagne, 1985). Studies with children have 
shown that they tend to ask more questions about stories with high levels of uncertainty 
or novelty, than about stories with low levels of uncertainty or novelty (Berlyne and 
Frommer, 1966). From an educational point of view, this finding suggests that 
motivation to learn can be increased by exposing students to situations which may cause 
them to experience doubt or uncertainty. 

Berlyne (1961) suggests several types of concepnial conflicts which may be used 
to arouse curiosity: 

• Doubt is a conflict between the tendencies to believe and disbelieve. The 
use of hard to believe ideas will most likely cause arousal and the resultant 
exploration and information gathering aimed at removing the conflict. 
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• Contradiction is a conflict between those attitudes and beliefs held by the 
student and opposite statements proposed by the educator. 

• Perceptual incongruity is an internally inconsistent sensory input 

• Conceptual incongruity is information about events and objects which 
contains incompatible elements (e.g.. Alaskan Palm trees). 

• Confusion or ambiguity stems from unclear ideas or incom^'lete 
information with behavior aimed at simplifying the ambiguous 
information. 

• Novelty contains a degree of unexpectedness and uncertainty that 
stimulates interest and curiosity. 

It should be noted that a very high level of arousal may produce poor 
performance. Optimum performance is achieved with a moderate level of arousal. 
However, this level may vary from individual to individual. 

Cognitive dissonance. Like curiosity, cognitive dissonance is triggered by a 
conflict between ideas (Davis, 1983; Festinger, 1957). However, the term curiosity tends 
to be used to describe the impetus to explore; whereas the term cognitive dissonance 
tends to be used to describe the impetus to rescue a conflict by changing one's attimdes 
or opinions. For example, cognitive dissonance may arise when students' opinions differ 
from those of their parents, teachers, or society as a whole. When these conflicts occur, 
the tendency is to change attitudes and opinions toward credible sources which are 
respected, and away from sources which are not respected. The implication of cognitive 
dissonance for education is that the teacher or intelligent mtoring system designer should 
strive to maintain credibility and respectability. 

Desire for competence. People seem to have an iimate desire to develop skills 
and acquire knowledge (Davis, 1983; White, 1959). This desire is especially evident in 
young children, who often want to do things (like crawling or eating) for themselves. 
The implication for educators is that this desire should be encouraged, not stifled, 
whenever possible. 

Need for achievement. The concept of a need for achievement helps explain the 
discrepancy between a student's abUity and his actual classroom performance (Davis, 
1983). This concept can be further broken down into autonomously oriented 
achievement modvation, which is based upon bettering one's previous performance; and 
social-comparison oriented modvadon which is based upon comparison with others. 
Theorists suggest that autonomous achievement modvadon develops early in a person's 
life and influences achievement as early as the second grade while social comparison 
achievement develops later-in about the fourth or fifth grades. Regardless of the 
student's age, each possesses simultaneous needs for achievement and failure avoidance. 
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McClelland (1985) points out that these needs are learned through environmental 
influences, and that the varying strength of these needs results in different behavior 
patterns in the classroom. For example, students with a strong achievement need will 
most likely prefer tasks of intermediate difficulty; whereas those whose fear of failure is 
greater than their need to succeed will most likely prefer either tasks that are very easy, 
guaranteeing success, or tasks which are obviously very hard for anyone. 

These findings suggest that students with a low need for achievement and/or a 
high fear of failure should be identified and encouraged to set goals for themselves, and 
to accept moderate risk. 

Attributions. In the 1970's and early 1980's a great deal of research was devoted 
to attribution theory (Weiner, 1979, 1980), which assumes that all persons are rational 
and have a need to understand their environment. This theory assigns a central role to 
the content of thought in motivation-specifically, whether the cause of success is 
attributed to internal factors, especially ability and effort, or to external factors such as 
task difficulty, luck or teacher bias. Some of the more noteworthy findings of this 
research as summarized by Davis (1983) are as follows: 

• Young children tend to attribute success to high ability, high effort or 
both, which in turn increases their feelings of self-worth and strengthens 
their achievement motivation. 

• Success in competitive situations enhances self-esteem and strengthens 
attributions of personal high ability. 

• Attributions become more logical as children get older. Specifically, as 
children get older they tend to attribute their successes to effort and ability 
and to recognize that it takes more ability to accomplish difficult tasks as 
opposed to easy tasks. 

• Minority i ndents tend to focus on external attributions such as task 
difficulty and luck, which are a form of learned helplessness. 

• Students are more likely to attribute success to effort and ability when 
they are engaged in individualized learning programs, where task 
difficulty is matched to the student's ability level. 

• Attributions of college students seem to have a self-serving, ego 
protecting function in that successful students attribute their success to 
ability and good performance while those who perform poorly focus on 
external factors. 

• Those who attribute success to either task ease or high ability have higher 
expectancy for success than do those who attribute success to luck or 
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effort. 


• When students believe their success is due to a stable characteristic, they 
are more confident of future success because they believe the 
characteristic will exist in the future. In addition, when students perceive 
that their effort is a stable characteristic, they are more likely to raise their 
expectations for future success and will keep putting forth an effort. 

The main implication of attribution theory is this: apart from a student's ability, 
his or her pattern of attributions can have a strong influence uj)on achievement. If 
students attribute their past failures to lack of effort, they are likely to try harder; whereas 
if they attribute their failures to lack of ability, they are more likely to give up. 
Similarly, if failure is attributed to bad luck, an individual is more likely to keep trying 
because things could change; whereas if they attribute failure to task difficulty, they are 
more likely to give up when they do not think task difficulty will change. Thus it is 
paramount that educators emphasize and encourage internal attributions such as effort as 
the route to success and point out that lack of effort and a focus on external factors leads 
to failure. 

Locus of control. The concept of locus of control is closely related to attribution 
theory (Davis, 1983). People who tend to attribute success and failure to external factors 
such as luck or the teacher, are said to have an external locus of control. People who 
attribute successes and failures to their own ability or effort are said to have an internal 
locus of control. However, attributions may vary from situation to situation, whereas 
locus of control is more of a stable personality trait. That is, a student who consistently 
attributes her successes to luck probably has an external locus of control. Students with 
an external locus of control tend to be less motivated to make an effort in academics. 
Teachers need to be aware of these students and to encourage them to have higher self 
expectations. 

Self-efficacy. Self-efficacy is used to refer to an individual's judgments about his 
or her ability to organize and implement actions in specific situations that may contain 
novel, unpredictable, and possibly stressful features (Schunk, 1984). Students' feelings 
of self-efficacy may affect their motivation to undertake a task, and the amount of effort 
and persistence they expend in completing it (Bandura, 1989). Meece, Blumenfeld, and 
Hoyle (1988) report that students who are concerned about their ability tend to use effort¬ 
minimizing strategies such as seeking frequent help, copying answers, and guessing at 
solutions. Reducing effort appears to be a defensive strategy used to protect feelings of 
self-worth, and to avoid the negative implications of low ability in the case of poor 
performance. By contrast, students who feel more efficacious not only exert more effort, 
but also have higher levels of intrinsic interest (Bandura and Schunk, 1981). 

Individuals' perceptions of self-efficacy generally increase if they: 1) directly 
experience mastery of a goal; 2) observe others like themselves succeeding by 
perseverant effort; and 3) are persuaded by others that they are capable of success 
(Bandura, 1989). Teachers should be aware of some of the behavioral, signs of low self- 
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efficacy, and be prepared to encourage students with low self-efficacy. 

Nature of the goal. Certain properties of goals may also have an effect on 
students' motivation. For example, goals which are proximal (i.c., close at hand) tend to 
be more motivating than goals which are distal (i.e., far off). A study by Bandura and 
Schunk (1981) found that students encouraged to set proximal goals for themselves 
performed significantly better and had higher levels of self-efficacy for a particular task 
than students encouraged to set distal goals or who were not encouraged to set goals at 
all. Also, goals which are more specific are more motivating than general goals. And 
finally, assuming requisite skills, individuals tend to expend greater effort on difficult 
goals (Schunk, 1990). 

Observational learning. Individuals tend to observe and imitate the behaviors of 
others, particularly those they respect or identify with (E. Gagne, 1985) Students will 
often watch and learn from the behavior of more successful or popular students, or from 
the behavior of their teachers or parents. Teachers can take advantage of this tendency 
by trying to get the more popular students to be academically involved, and by modeling 
appropriate learning strategies themselves. 

Orientation toward learning. Some motivation researchers (e.g., Dweek, 1986) 
have made a distinction between mastery and performance orientations toward learning. 
Individuals with a mastery orientation seek to increase their competence-'to master 
something new. Individuals with a performance orientation seek to gain favorable 
judgments about their competence or to avoid unfavorable judgments. 

Dweek notes that these patterns can have profound effects on cognitive 
performance. A mastery orientation is characterized by challenge seeking and 
persistence in the face of obstacles. A performance orientation is characterized by 
challenge avoidance and low persistence in the face of difficulty (Dweek, 1986). 

From a teaching point of view, challenging tasks are often the best way to 
develop students' capabilities. Students with performance orientations are less likely to 
benefit from challenges. Teachers should help such students to develop a mastery 
orientation by encouraging them to pursue challenges and then teaching them to attribute 
failures to effort or strategy, rather than ability (Dweek, 1986). 

Gender. Bright girls, as compared to both bright boys and less bright girls, seem 
to demonstrate shakier expectations, have lower preference for novel or challenging 
tasks, more frequently attribute failure to lack of ability, and more frequently debilitate in 
the face of confusion or failure (Licht & Dweek, 1984; Stipeck & Heffman, 1980). 
Lewis and Cooney (1987) also state that female students tend to perform best when the 
comparison is based upon their own performance as opposed to males who tend to 
perform best under situations when comparison is in a social context. It is not clear why 
this tendency exists, but in any case, teachers should be aware of it, and be prepared to 
encourage girls accordingly. 
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Domain. Motivation can also vary with domain. A study by Young. Arbreton 
and Midgley (1992) examined modvation and cognidon in four academic domains; 
math, English, science and social studies. They found that among the different domains, 
students exhibit different goal orientadons (performance vs. mastery orientation) and 
strategies (surface vs. deeper level of cognitive strategies). Students are more likely to be 
mastery oriented in both math and science in comparison to either English or Social 
Studies. Students' reported use of deeper processing strategies was significantly higher in 
English than in math, science and social studies, and deep processing was used 
significantly less in social studies than in all other content areas. In all areas there is a 
significant negative relation between mastery orientation and surface processing 
strategies and between performance orientation and deep processing strategies. 

Students were most likely to hold a performance-focused orientation in social 
studies than in other content areas and were least likely to hold such an orientation in 
math and science. Finally, students were significantly more likely to use surface level 
strategies in social studies than in all other subjects, and they were less likely to use these 
types of strategies in science. 

In sum, motivational techniques should take domain specificity into account. 
Knowledge of which types of strategies are most likely to be used in specific content 
areas can be used to guide the selection of techniques. 

Training. There is evidence to suggest that students can be taught to change 
demotivating or debilitating thought patterns. Klein and Freitag (1992) utilized Keller’s 
ARCS method to design a booklet which provided students with information about 
motivation, examples, practice and feedback on how to make instruction relevant. The 
booklet was designed to be self-paced. Results indicate that students who were given 
self-motivational training significantly rated instructional tasks as having more relevance 
than students not given this training. These results were true for both immediate and 
delayed training effects. 

Tools for Assessment of Motivation 

Harter (1981) developed the Scale of Intrinsic Verses Extrinsic Motivation in the 
Classroom. This scale assesses the degree to which children manifest an intrinsic interest 
in learning, view themselves as being curious, and show a preference for challenging 
work or for opportunities to master material independently. Ames and Archer (1988) 
developed a questionnaire which examined the areas of goal orientation, learning 
strategies, task challenges, attitudes towards class, causal attribution and perceived 
ability. In addition, many personality tests include subscales for various aspects of 
motivation; and tests exist for some of the factors listed in the previous section (e.g., 
locus-of-control). However, there is no widely accepted instrument for a thorough 
assessment of motivation in the classixxim. 
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Evaluation of the Research 


To enable an intelligent tutoring system to keep students motivated, developers 
need to know what to monitor, how to monitor it, and how to respond with appropriate 
interventions. The research literature on motivation at least partially answers the what to 
monitor requirement. However, it is less helpful in prescribing how to monitor 
motivation and how to keep it high. 

In terms of what to monitor, the literature has identified a number of factors 
which affect motivation. However, the terminology is spongy. Often different 
constructs, such as self-efficacy and the idea of having a performance orientation toward 
learning, seem to be describing the same thing. Assessment would be more efficient if 
the factors were more distinct, and ITS developers knew which factors influence 
motivation the most. 

Without knowledge of which factors influence motivation the most, it is 
impossible to know how to monitor it. The fact that there is no widely accepted 
instrument for measuring student motivation available confirms this. Such an instrument 
needs to be developed. Perhaps this instrument could take advantage of some of the test 
presentation and data collection capabilities of the computer. 

Finally, the literature gives little guidance about how to sustain and/or increase 
motivation in specific situations. There is plenty of general advice, such as engage 
curiosity and encourage effort rather than ability attributions. However, there is 
relatively little precise guidance about what to do in, for example, a situation where a 
student has a low need for achievement, low self-efficacy, and an internal locus of 
control. Most studies of factors affecting motivation examine only one factor at a time. 
There needs to be more research on how the factors interact. 

There also needs to be more research on ways of sustaining motivation over a 
long period of time. For example, some techniques, such as those which pique curiosity, 
may affect state motivation, but not trait motivation. Perhaps in some cases, motivating 
and demotivating factors effectively cancel one another out in the short term, but have 
lingering effects in the long-term; for example, a student with a high need for 
achievement (e.g., to please his parents), but low self-efficacy may perform well in 
school, but ultimately fail to reach his full creative potential. Also, use of some 
techniques for increasing motivation (again, such as those which pique curiosity) may 
lose their effect over time, as students get used to seeing them. 

Applicability to Intelligent Tutoring Systems 

The current research yields broad prescriptions which could help to increase 
motivation in many cases. It is probably infoimative enough for traditional computer- 
assisted instruction, which relies mostly on the developer's ability to anticipate the 
student's needs anyway. However, it is probably not infonnative enough for intelligent 
tutoring systems, which offer the capability of targeting instruction to specific student 
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needs. Intelligent tutoring system development (as well as education in general) would 
benefit from a unifying theory of motivation, which explains what factors affect 
motivation and the extent to which they add to or mitigate each other's effects. 


Areas for Assessment 

A major goal of intelligent tutoring systems is individualization of instruction. 
The hope is that by tailoring instruction to individual students’ needs, all students can 
achieve certain minimum learning requirements, and every student will have the 
opportunity to fully develop his or her potential. 

Of course, individualization requires some knowledge of the individual. One 
approach to individualizing instruction involves representing individual students' 
knowledge as it develops (Shute, 1991). This approach generally involves developing 
bug libraries or other student models (Brown & Burton, 1979). iMthough this can be an 
effective approach, it is costly to implement In addition, effective instructional decision 
making may involve more than knowing what the student knows. It may also involve 
knowing about certain personality characteristics of the learner, such as self-efficacy, 
motivation, and cognitive style. 

Another approach to individualizing instruction involves the assessment of 
incoming knowledge, skills, and personality characteristics (Shute, 1991). These 
incoming characteristics can then be used to determine appropriate instructional 
treatments. 

The two approaches arc not mutually exclusive. In fact, they could be 
complementary. Pre-assessment can be used to determine the overall approach to 
instruction, including the structure of teacher-student interactions, and the manner and 
sequence in which new information will be presented. Smdent models can be used to 
monitor student understanding play-by-play, as it were, which can be helpful for 
decisions such as what type of teaching action to take next, or what topic to cover. 
Student models may also be used to make adjustments to the approach initially 
determined by pre-assessment. 

This section will focus on identifying the types of information needed for 
assessment 

Summary and Evaluation of Research 

There are two approaches to identifying the types of information needed for 
assessment One way is to identify the information expert human teachers attend to 
while interacting with their students. Examining what teachers look for and how they 
use this information could provide clues about the types of information that a computer 
should assess both before and during instruction. 
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A second approach is to study how particular psychological constructs (e.g., 
intelligence, motivation, cognitive style) affect learning. Assessing aptitudes or 
personality traits could then allow the computer to target instructional treatments to 
particular types of individuals. The following sections briefly review studies using both 
types of approaches. 

Studies of teachers . One would expect that teachers utilize many different tyjjes 
of information in making instructional decisions. However, if this is so, it is apparently 
not at a conscious level. Studies of teacher behavior have shown that teachers have 
difficulty identifying what variables they attend to and what principles guide their 
instructional decision-making. Only two variables are routinely identified by teachers as 
having an impact on their decision making (ability and motivation) and even thete 
constructs mean different things to different teachers (Como & Snow, 1986). 

What teachers do attend to is their mental teaching scripts (Borko & Shavelson, 
1990) These scripts consist of well-established routines for various teaching aedvities 
(one commonly used script, for example, is the pattern of structuring, soliciting, 
responding, and reacting). Researchers have found that in general, classroom teachers 
only engage in interaedve decision-making when they are forced by problems or 
unexpected events to abandon their scripts (Borko & Shavelson, 1990; Shavelson & 
Stem, 1981; Clark & Yinger, 1979). 

The needs of intelligent tutoring systems development have spawned a number of 
studies of human mtors in one-on-one situations (e g., Merrill, Reiser, Ranney, & 
Trafton, 1992; Schoenfeld, Gamoran, Kessel, & Leonard, 1992; Fox, 1991; Galdes & 
Smith, 1990; Yee, 1989; Littman, Pinio, & Soloway, 1985; Collins, W^amock, & 
Passafiume, 1975). These studies have been much more successful at identifying the 
types of infomtation attended to by human tutors. However, although these studies make 
the logic behind these tutors' instnictional decision-making more evident, it is not clear if 
the decisions made were the best in an absolute sense or simply the best the tutors studied 
could think of 

In conclusion, deriving information from human teachers may not be the best 
approach to determining what information to assess. First, in some cases it may be 
difficult to get them to precisely identify what types of information they use. Second, 
even if it is possible to discover this information, it is still necessary to evaluate whether 
or not this information is 1) helpful in instructional decision making; and 2) feasible for 
intelligent tutoring system implementation. 


Studies of psychological constructs . The other approach to identifying 
information for pre-as.sessment is to study the impact of various psychological constructs 
on teaching effectiveness. For example, Shute (1991) found an interaction between an 
exploratory learning style and type of learning environment. Specifically, high 
exploratory learners learned better from an inductive learning environment; whereas low 
exploratory Icarr.crs learned better from a structured, explicit learning environment. 






Tabic 4 lists some pre-assessment variables which have been frequently 
mentioned in discussions of adaptive instruction. It also gives their definitions, what 
research has shown about their relationship to learning, and a rough assessment of their 
applicability for implementation within intelligent tutoring systems. 

The table indicates four variables judged to be particularly relevant to 
implementation in intelligent tutoring systems. Some comments on these variables 
follow. 


Intelligence is consistently the best predictor of academic success (Como & 
Snow, 1986; Hakstian & Gale, 1979). Likewise, educational history is helpful in 
determining at what level instruction should begin. Students with high levels of prior 
knowledge or inteliigence could be bored if instruction is too easy; students with low 
levels of both may become frustrated if instruction is too hard. Much research has been 
conducted in these areas, especially to determine what types of instruction are most 
beneficial for students of higher or lower ability. However, there seems to be some 
disagreement over whether measures of past achievement or measures of aptitude are 
better for this purpose. 

The research also strongly indicates that self-efficacy is related to performance. 
Timm erman (1989) found that increases in self-efficacy are relatul to improved 
performance even while controlling for ability level. Other researchers (Carlson and 
Grabowaki, 1992; Bandura, 1989) have found that domain-specific measures of self- 
efficacy provide more accurate assessments. Self-efficacy is particularly appealing for 
implementation within intelligent tutoring systems because it can be reliably measured, 
and it can be improved through various interventions. 

There are many dimensions to the construct referred to as cognitive style (e.g., 
field independence/dependence, reflectivefimpulsive, scanning/focusing, 
leveling/sharpening); however, not all have well-documented effects on learning. 
However, research on field independence/dependence has been going on for a longer 
time than research for the other dimensions, and tends to be of a higher and more 
consistent quality. Most of this research suggests that field independence/dependence 
can be a valuable variable for pre-assessment, and that individuals from the two groups 
do pick up information differently (in terms of how they perceive the world and how 
they leam). However, researchers need to take a closer look at people who score at or 
close to the mean. At least one study (Meng & Del, 1991) suggests that these field- 
intermediates differ from the extremes, and that they need different instruction (rather 
than something in-between) from the other two groups. 

In conclusion, some clear connections have been established between individual 
constructs and performance which should help in deciding how to individualize 
instruction. However, in most cases the influence of particular constructs has been 
smdied in isolation. There need to be more studies which examine the interactions 
between specific constructs and instructional treatments, or between combinations of 
constructs. 
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It would be ideal if the computer was able to figure out for itself which variables 
seem to have the greatest impact on learning for each student. This would be an 
excellent area for the application of machine learning concepts. 

Applicability to Intelligent Tutoring Systems 

Research about assessment variables is very applicable to intelligent tutoring 
systems development. However, knowing what needs to be assessed is just one part of 
the problem. Another part is to figure out how to assess these variables on the computer, 
and with minimal loss in instructional time. Yet another part is developing principles for 
how the assessment information will be used in instruction. It is important to establish 
that the uses of the information will be worth the trouble of gathering it 

Even if perfect adaptation of instruction to the student (so as to always maximize 
performance) were possible, it is not clear if it is desirable. There is some evidence to 
suggest that mismatching educational treatments to learner characteristics may stimulate 
flexibility and creative thinking (Messick, 1984). It is important to remember that 
judgments about educational programs ultimately require decisions to be made about 
social values. 
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Table 4. Potential Variables for Pre-Assessment 


1 Varitble 

Definition 

Instructional Decisions Impacted 

Applicability 

H Educational hisioiy 

Background knowledge in the 
domain to be taught 

What to teach; whether or not to use a 
direct inslnictiQn ^iproacfa (Como & 
Snow, 1986) 


Intelligence 

Score on various recognized 
measures of IQ 

What to teach; whether or not to give 
cognitive strategy training; whether or 
not to use discovery learning; what 
degree of learner control to allow 
(Como & Snow. 1986; Frederickaon. 
1984) 

• ** 

1 Self-efficacy 

An individual's personal 
judgment of performance 
capabilities in a given domain 
(Schunk. 1984a) 

What types of instructional motivators 
to use (Schunk, 1984b) 


Gignitive style - Field 
independent, 
dependent, or 
intermediate 

Degree to which an individual 
can distinguish a single Hgure 
within a larger, more complex 
figure. Often determined by 
score on Group Embedded 
Figures Test (Wtldn, Oltman, 
Raskin, & Karp, 1971) 

Whether or not to give kaming 
strategies training (on Uttks such as 
taking notes); to what degree should 
structure of a lecture/lesson be made 
clear; what type of advance organizer 
to provide (Meng & Del. 1991; Frank, 
1984) 

• •• 

Motivation 

That v'^ich gives direction and 
intensity to behavior (E. 

Gatme, 1985) 

Whether or not students need to be 
taught self-motivation strategies 
(Klein & Keller. 1990) 

• • 

Test anxiety 

Anxiety about taking tests 

Whether or not to give training on 
how to deal with anxiety; whether or 
not to provide competitive learning 
situatioiu (Naveh-Benjamin. 1991; 
Weinstein & Mayer, 1986) 

«• 

Cognitive style - 
reflective or inqmlsive 

Degree to which an individual 
thinks over an action before 
taking it Generally measured 
by Matching Familiar Figures 
Test (Kagan, 19ti5) 

Whether or liot to give training in 
enective problem-solving strategies; 
whether or not to give training in 
arudytic reading strategies; whether or 
not practice using a random schedule 
(He^eU AUen, & Stone, 1991; 

Waiczyk & Hall, 1989; Jelsma, Van- 
Merrienboer. & Jeroen. 1989) 

• • 

Locus of control - 
internal, external 

Degree to which individuals 
feel they have control over 
their environment. 

Whether or not to use direct 
insDuction; whether or not to 
encourage development of intenial 
locus of control (Klein & Keller, 

1990) 

* 

Learning style - 
accomodator, diverger, 
converger, or 
assimilator 

Degree to which an individual 
prefers active or reflective, and 
concrete or abstract modes of 
learning. Generally measured 
by Kolb's Learning Style 
Inventory (Kolb. 1984). 

No significant effects were foimd in 
studies of the relationship between 
learning style and effective learning 
(Billings & Cobb, 1992; Karrar, 1991; 
Larsen. 1992) 



*** Very applicable, easy to implement 
** Applicable, may be somewhat difficult to implement 
* Could be applicable, uncertain about implementation 


35 




































Use of Questions 

Because intelligent tutoring systems do not have eyes or ears, the best way for 
them to get information about students is to ask. Therefore knowledge about the 
appropriate use of questions is especially important for ITS development. 

Definition 

Questions can have several functions. The most common function is to get 
information, which in instructional contexts generally means information about what the 
student knows. However, questions can also be used to promote higher-level cognitive 
processes, to gain attention, or as an indirect command (e.g., "could you pass me the 
salt?"). The focus in this section will be on instructional uses of questioning. 

Review and Evaluation of Research 

The review included literature on question taxonomies and formats. There are 
numerous taxonomies which can be used for classifying questions (see Gall, 1970, and 
Graesser, 1992, for examples), but the most well-known one by far is Bloom's taxonomy. 
This taxonomy consists of the following levels: Knowledge, Comprehension, 
Application, Analysis, Synthesis, and Evaluation (Davis, 1983). These six levels 
describe progressively higher levels of cognitive processing. Figure 4 lists some sample 
questions for each level of the taxonomy. 


Knowledge - What is the formula for computing the standard area of measurement? 

Comprehension - Examine the graph and determine how many items must be added to a 
SO-item test to increase its reliability from 0.60 to 0.80. 

Application - Compute the standard error of estimate for a test having a correlation of 
0.70 with a criterion having a standard deviation of 10. 

Analysis - Differentiate between a classroom achievement test and a standardized 
achievement test in terms of what each measures and how each is used. 

Synthesis - Formulate a theory relating interest to personality, citing appropriate 
supporting research evidence. 

Evaluation - Evaluate the criticisms of Ralph Nader and Allen Naim concerning the 
Scholastic Aptitude Test (SAT). 


Figure 4. Sample Test Items for Each Level of Bloom's Taxonomy (from Aiken, 
1991). 
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Although Bloom's taxonomy neatly expresses the hierarchical nature of 
knowledge, it does not capture the full range of reasons why a question might be asked. 
Gall (1970) identifies some worthwhile questimi types which are not covered: 

• questions which cue students to elaborate on a weak response to a 
previous question (e.g., "Gut you tell me more?" or "What do you mean 
by that?") 

• questions which stimulate a student's curiosity or sense of inquiry (e.g., 
'What would you like to know about dinosaurs?") 

• questions which guide a student's learning of a Rkill (e.g., "What do you 
think we should do next?") 

Intelligent tutcaing systems could potentially use all of these question types. 
However, the research lacks guidance about when to use which type. There needs to be 
more research on how to sequence question types (e.g., how do you know when to start 
asking questions at a higher level?), and when to use the various types of questions. 

The literature on question formats covers when and how to use written formats 
such as true-false, multiple choice, and essay questions (Aiken, 1991; Davis, 1983), as 
well as effective classroom questioning strategies (Omstein, 1988; Otto, 1991). 
Although this literature contains helpful guidance about how to ask questions, much of it 
is geared toward classroom situations rather than computer-based teaching situations. 
For example, most classroom questions are either oral (i.e., in the context of a lesson) or 
paper-and-pencil based. Computers are not good at dealing with oral questioning, and 
they ate capable of much more than the standard multiple-choice or true-false formats, 
(^estioning on the computer needs to be looked at in a completely different way from 
questioning in the classtootiL 

Applicability to Intelligent Tutoring Systems 

The problem with intelligent tutoring systems is that they carmot deal with open- 
ended question formats as easily as human teachers can. During instructional 
presentations, human teachers routinely obtain information about how much students 
understand simply by asking them to answer questions or provide explanations. With an 
intelligent tutor, the same pitxess would be cumbersome in terms of both data input (the 
student would probably have to type) and answer analysis (computers' natural language 
processing capabilities are still very limited). In addition, human teachm can gather 
some information simply by observing their students; whereas an intelligent tutoring 
system would have to ask a student in order to get input 

Although this problem is not insurmountable, further research needs to be 
conducted on ways of presenting and assessing knowledge via the computer. For 
example, one capability Aat computer-based tutors have that human teachers lack is the 
ability to provide interactive games and simulations. For certain te^hing situations. 
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perhaps students can be required to perform continuously. This would allow the 
computer to gather the data it needs, and it can interrupt when the student appears to need 
help (or to praise the student). From a teaching point of view, this would represent a 
paradigm shift—from a format where the teacher does most of the telling with pauses for 
student performance, to a format where the student does most of the performing, with 
pauses for instructions from the teacher. 


Definition 

Broadly speaking, instructional feedback can be anything a teacher does in 
response to a student comment or action. Even if a teacher does nothing, his or her 
inaction can be construed as a type of feedback, because it sends a message to the student 
(e.g., "You're doing okay so far. Keep going."). Or if a student is in the middle of 
performing a task, giving feedback may involve first interrupting her, then correcting or 
praising her. 

In practice, however, people usually think of feedback somewhat more narrowly. 
In studies of feedback in traditional computer-based instruction, for example, feedback is 
typically viewed as how the computer responds after the student has answered a question 
(Wager & Wager, 1985). The pattern is: computer asks question, student answers, 
computer gives feedback. Perhaps this is because traditional computer-based 
instructional systems are only capable of giving feedback in this manner. 

Because intelligent tutoring systems have more sophisticated capabilities than 
traditional computer-based instruction, US developers and researchers should take the 
broader view of feedback in planning how to use it in their systems. However, findings 
from research which takes the narrower view arc still very applicable. 

Purpose 

Feedback can serve two functions during instruction (Wager & Wager, 1985). 
First, it can motivate the learner. Second, it provides information about the correctness 
of the learner's responses. It should also be noted that poorly constructed feedback may 
produce the opposite outcomes. 

Summary and Evaluation of Research 

The review of the literature focused primarily on how various elements of 
feedback (such as amount of detail or timing) affect performance, anxiety, confidence, 
and learning efficiency. These findings are summarized in Table 5 (pages 41-45). The 
left-most column lists elements of instruction which could be included in or with 
feedback presentation, such as: 
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Level of detail 

• no feedback 

• knowledge of results (KOR) - whether a response is correct or incorrect 

• knowledge of the correct response (KCR) 

• answer until correct (AUQ - the student has to keep answering the 
question until he gets it right 

• elaborated - includes explanation. The content varies, but usually 
involves either telling the student why her answer was wrong or right, or 
giving her information which will help her to get the answer right in the 
future. 

• natural - results of a smdent's actions are displayed automatically, as in a 
simulation 

• adaptive - any of the above, depending on certain situational criteria 

Timing relative to the student's response 

• immediately after 

• delayed - amount of delay may range from seconds to days 

• when requested - learner control of feedback 

With other instructional support available (e.g., lesson material) 

• available 

• not available 

With performance feedback (e.g., scores) available 

• available 

• not available 

Personalization (e.g., addressing the student by name) 

• available 

• not available 

The top row of the table lists factors which might be affected by the presence or 
absence of the elements in the left-hand column. These include task performance, non¬ 
performance factors such as anxiety and confidence, and overall lesson efficiency (i.e., 
the time it takes to achieve a certain performance result). The cells in the table list 
findings from the literature about how the elements of feedback affect these factors. 

Some general comments on the findings: First, the research on the required 
amount of detail in feedback is not very conclusive. It seems safe to conclude that any 
amount of feedback is better than none at all in terms of improving performance. But it 
is not clear how the various levels of detail compare to one another. In general, more 
information does not appear to hurt student performance (Salas & Dickinson, 1991). But 
if the additional information does not actually get used, then including it may be a waste 
of both lesson time and development time (Qariana, Ross, & Morrison, 1992). 

Ultimately, the optimal amount of information in feedback mgy depend on the 
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learner. Low ability or low achieving learners may benefit from more explanation in 
feedback (Collins, Gamine, & Gersten, 1987); however, they also need to be taught how 
to utilize feedback information effectively. High ability learners seem to use knowledge 
of results feedback more effectively than low ability learners (Hansen, 1974). But too 
much information for these types of learners may inhibit them from learning to reason 
for themselves (Clariana, Ross, and Morrison, 1992). 

Surprisingly little is known about how feedback form or content affects learning 
(Wager & Wager, 1985). There are at least three reasons for this. First, feedback can 
take many different forms, ranging from simply stating that an answer is correct or 
incorrect, to revealing the correct answer, to allowing students to keep trying till they get 
the correct answer, to providing explanations about why certain answers are right or 
wrong. Explanations can be long or short, reviews or restatements, text-based or 
pictorial. Without specifying the form of feedback used, it is impossible to speak 
meaningfully about the effects of feedback. 

One interesting example of how the content of feedback affects how a study 
should be interpreted is a study by Hansen (1974) on the effect of feedback on state 
anxiety. Hansen predicted that use of feedback should reduce learners’ levels of state 
anxiety, and that high anxiety state subjects would make fewer errors under the feedback 
condition than under the no-feedback condition. He found that in general, feedback did 
decrease levels of anxiety. However, high anxiety subjects actually made more errors 
under feedback than under no-feedback conditions. It turns out the feedback condition 
used in his study consisted only of telling subjects if their responses were correct or 
incorrect (KOR feedback). It is easy to imagine that an already anxious subject might 
become more anxious after missing a question and not knowing why. Therefore Hansen’s 
findings about the effect of feedback on state anxiety should be qualified to apply for 
KOR feedback only. 

The second reason why little is known about how the content of feedback affects 
learning is that there are many factors which must be considered in evaluating the effects 
of feedback. These include the difficulty level of the material to be learned, the 
characteristics of the learners, and the context of the study (i.e., lab or school). Why 
feedback is found to be effective or ineffective often depends on the manner in which it 
was evaluated. For example, many studies of feedback are conducted in a laboratory 
environment (Kulik & Kulik, 1988). These types of studies have consistently shown that 
feedback results in better retention if it is given after a delay of as much as 24 hours 
(Sturges, 1978; Rankin & Trepper, 1978; Sassenrath, 1975). However, in laboratory 
studies, there is usually no additional instruction, and the treatment questions are 
identical to the post-test questions. And in fact, in studies conducted in more realistic 
learning situations, students almost always perform better when they receive immediate 
rather than delayed feedback (Kulik & Kulik, 1988). 

Finally, the terminology used in feedback research varies from study to study. 
Even the definition of the word feedback varies. For example, some studies consider 
stating an answer to be correct or incorrect to be feedback, while others treat it as a no 
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feedback condition. Although variations in terminology may not lessen a study's 
contribution, they do make the findings less accessible. 

Applicability to Intelligent Tutoring Systems 

The existing research on feedback is clearly applicable to intelligent tutoring 
systems. However, much o€ it was conducted wi^ national paper-and-pencil type 
tests. Even the research that was done using the computer tends to be with traditional 
text-based questitm fomiats. There needs to be more research on effective uses of the 
types of fe^back uiuque to intelligent tutoring systems. That is. there needs to be more 
research on ways of t^oring feedback to the learner, since ITSs have the capability of 
being able to individualize instruction. And there also needs to be more research on 
effective use of feedback for the types of assessment capabilities (i.e.. uses of games and 
simulation to assess performance) which are uiuque to the computer. 
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Table 5. Feedback Research Summary. 
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mty increase depth of processing 
for that item (Clariana, Ross, & 
Morrison, 1992)._ 





























. Feedback Research Summary (cont.) 

Comments on FindinRS 

The main Ending seems to be 
that elaborated feedback is not 
necessarily either efficient or 
effective. Helpful elaborated 
feedback, whatever its content, 
provides information which the 
student needs and can use Just 
because there are more words on 
the screen does not mean that 
they will be processed 
meaningfully. 

Effective use of naniral feedback 
probably depoidt on the nature 
of the subject being Uught and 
the ability/background of the 
learners. Park and Oittelman 
(1992) suggest that naniral 
feedback is likely more 
beneficial for high ability 
students, while elaborated 
feedback is more beneficial for 
low ability students. 

Impact on Learning 
Efficiency 

No significant difference in 
lesson completion times between 
elaborated and KCR treatment 
groups (Collins, Catnine, & 
Gersten, 1987) 

Significantly more efficient than 
KOR feedback in terms of time 
required to complete a lesson. 

Impact on Non- 
Performance Factors 

Subjects receiving elaborated 
corrections were significantly 
more confident about their ability 
to perform the trained task than 
students receiving KCR feedback 
only (Collins, Camine, & Gersten, 
1987) 

No research idenliEed. 

Table 5 

Impact on 

Performance Factors 

Merrill (1987) found no significant 
difference between elaborated 
feedback emphasizing key attributes 
of concepts to be learned and KCR 
feedback. However, Salas A 

Dickinson (1991) found that 
elaborated feedback which includes a 
nmemonic to help students remember 
key attributes of a concept was 
significantly more effective than 

KOR feedback, and the most effective 
form of elaborated feedback. Collins, 
Camine, A Gersten (1987) found 
elaborated feedback was significantly 
more effective than KCR feedback for 
learning disabled and remedial high 
school students. McKendree (1990) 
found that students receiving 
elaborated feedback about the goal 
structure of geometry problems 
petfonned beaer than those students 
receiving KOR feedback or 
elaborated feedback which explained 
why their answer was Incorrect. 

As effective as KOR or elaborated 
feedback when used for appropriate 
tasks and with learners who know 
how to use it (Park A Gittelman, 

1992). 
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Table 5. Feedback Research Summary (cont.) 
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Table 5. Feedback Research Summary (cont) 
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Table 5. Feedback Research Summary (cont.) 



(i.c., feedback which referred to them 
by name) performed significantly 
better than students receiving the 
same feedback, but not personalized. 























Learping Sfratggies 


Definition 

Learning strategies (sometimes known as cognitive strategies) are techniques that students 
themselves use to enhance their understanding and retention of new material. Different strategies 
are used for different aspects of the learning process or for different learning situations. 
Strategies for elaboration, for example, help students to establish links between the information 
to be learned and what they already know, thereby enhancing recall. Strategies for 
comprehension monitoring enable students to recognize when they do not understand something. 

Purpose 

The purpose of including learning strategies in instruction is to enable students to become 
independent, effective learners. Some students have their own learning strategies, and would not 
benefit greatly from learning strategies instruction. Others, however, require more explicit 
guidance. 

Summary and Evaluation of Research 

Learning strategies research can be divided into two categories: 1) studies which identify 
the types of strategies used by effective learners; 2) studies which examine factors which can 
affect the use of learning strategies; and 3) studies which evaluate the effectiveness of various 
learning strategy training programs. 

Strategy Identification Studies 

Strategy identification studies identify the types of learning strategies used by effective 
learners. Generally, these types of studies involve comparisons of older or more successful 
learners with younger or less successful learners. The comparisons in Table 6 are based on a 
review of the literature by E. Gagne (1985). 

Table 6. Comparison of Learning Strategy Lise by Learner. 


More Effective Learners 

Less Effective Learners 

Have strategies for differentiating between 
more and less important information, and 
will selectively attend to important 
information. 

Are unable to differentiate between ideas of 
varying importance. 

Have strategies for remembering new 
information. 

Lack strategies for remembering new 
information 

Use strategies for organizing information 
(e.g., clustering related items, 
summarizing) 

Do not organize information 

Monitor their own learning 

Do not monitor their own learning, or do not 
monitor it effectively. 
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Gagne also points out that effective learners not only have a >vidc repertoire of learning 
strategies, but also know when to use which ones. V“ry effective learners will constantly monitor 
their level of understanding and may change strategies accordingly. 

A variety of different nomenclatures exist for classifying the different types of learning 
strategies. One of the most comprehensive is by Weinstein and Mayer (1986), which lists eight 
categories. These categories, their descriptions, and example tasks to which they could be applied 
are listed in Table 7. 


Table 7. Types of Learning Strategies. 


Learning Strategy 

Description 

Application 

Basic rehearsal strategies 

Actively reciting or naming 
presented items during 
learning 

Memorizing lists of items such 
as names of presidents or state 
capitals 

1 Complex rehearsal 

1 strategies 

Reading material aloud, 
copying material, taking 
verbatim notes, underlining 

Memorizing material from a 
text passage 

Basic elaboration strategies 

Building internal 
associations between two or 
more items to be learned 

Memorizing foreign language 
vocabulary 

Complex elaboration 
strategies 

Integrating new information 
with information already 
known 

Paraphrasing, summarizing, 
developing analogies from new 
material 

1 Basic organizational 

1 strategies 

Clustering items into groups 
to remember them 

Memorizing items from a list 

1 Complex organizational 

1 strategies 

Developing outlines, 
concept maps, networks 

Identifying main ideas and 
supporting concepts in a 
passage 

1 Comprehension monitonr,^. 

1 strategies 

Establishing learning goals, 
assessing degree to which 
goals are being met, and 
modifying strategics as 
needed to meet goals. 

Selective attention during 
reading 

Affective and motivational 
strategies 

Changing thought patterns 

Reducing test anxiety, 
maintaining motivation, 
maintaining concentration, 
focusing attention 


Factors Affectine Use of l.eamine Stratecies 


Most people probably use learning strategics to some extent. There are several factors 
which seem ro l)c relatcrl to the degree to which individuals use learning strategies effectively. 
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These include: level of domain knowledge, use of cognitive monitoring, persistence in the use of 
less effective strategies, learning context, self-efficacy, level of intrinsic motivation, and lack of 
training or experience. These factors are summarized in Table 8. 

Effectiveness of Learning Strateev Training 

As stated in the previous section, training in learning strategy use does help students to 
use these strategies more often. This does not necessarily result in an improvement in academic 
performance, however. Factors which may influence the impact of learning strategies training on 
academic performance include: 

Learner ability. There is some evidence which suggests that learning strategies training 
may actually hinder higher ability students in some cases (Brown, Trathen, & Dole, 1992). 
Periiaps these students already have effective strategies of their own, and being instructed to use 
some other approach ccxifuses them. However, this does not necessarily mean that high ability 
students would not benefit from learning strategies training-just that it may need to be presented 
differently. 

Lack of domain knowledge. Knowledge of learning strategies may make students better 
learners, but it is clearly no substitute for domain knowledge (Weinstein & Mayer, 1986). Solving 
a problem requires both domain knowledge and general cognitive strategies (Kelley, 1992). 

Applicability of learmng strategies training to particular academic situations. It is not 
enough just to teach about learning strategies. Sometimes it is necessary to teach the right 
learning strategy at just the time it is needed, and to point out why it is an appropriate strategy to 
use. Younger or low ability learners, for example, require complete, explicit instruction that is 
situated in the context of the domain being stuped ((jampione, 1987; Loper & Murphy, 1985; 
Paris, Goss, & Lipson, 1984; Pressley, 1986). 

As long as learning strategies instruction is carefully targeted, there is reason to believe 
that it can have a positive impact on academic performance (Loper & Murphy, 1985; Miller, 
1985). 


In summary, the research on learning strategies use clearly demonstrates that learning 
strategies instruction can be very beneficial. It also lets us know the conditions under which 
learning strategies are most likely to be used, and factors which may influence their effectiveness. 
However, more research which explores the interactions between the various factors would help 
intelligent tutoring systems to better target learning strategies instruction to the needs of 
individual learners. In addition, research is needed which answers the questions such as how 
often should learners be exposed to learning strategies concepts, and to what extent these 
strategies be incorporated into domain instruction (as opposed to being taught separately). 
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Table 8. Factors Affecting the Use of Learning Strategies. 


Potential Factor 

Definition 

Effect on Learning 
Strategy Use 

Level of domain 
knowledge 

Student’s knowledge of 
domain to which the learning 
strategy is to be applied 

Students' degree of domain 
knowledge appears to have 
some effect on the types of 
strategies used (Gagne, 1985; 
Gamer, 1990). 

Use of cognitive 
monitoring 

Student's tendency to assess 
his or her level of 
comprehension 

If students do not realize 
when they do not understand, 
they will not know that they 
need to apply a learning 
strategy (Gamer, 1990; E. 
Gagne, 1985). 

Persistence in the use of 
less effective strategies 

Sub-optimal learning 
strategies, such copying text 
verbatim rather than 
paraphrasing or summarizing 

If less effective strategies 
work (albeit less efficiently), 
student may tend not to use 
more effective ones (Gamer, 
1990). 

Learning context 

Learning situation -e.g., 
classroom environment, 
domain being studied 

Smdents tend to use more 
effective learning strategies in 
classrooms wnich emphasize a 
mastery learning approach 
(Ames and Archer, 1988). 

Also some evidence suggests 
that students may use 
different learning strategies 
for different subjects (Young, 
Arbreton, & Midgley, 1992). 

1 Self-efficacy 

Assessments of one's own 
ability to perform in novel or 
difficult situations. 

High perceptions of self- 
efficacy are consistently 
correlated with greater and 
more efficient strategy use 
(Bouffard-Bouchard, 1990; 
Pintrich & De Groot, 1990; 
Bandura, 1989). 

Level of intrinsic interest 
in domain being studied 

The extent to which the 
student is motivated to learn 
the subject for its own sake 

Intrinsic interest is correlated 
with strategy use (Pintrich & 

De Groot, 1990). 

Training and/or experience 
in cognitive strategy use 

The extent to which the 
student has been taught to use 
and practiced using cognitive 
strategies 

Training helps students to use 
strategies more (Weinstein & 
Mayer, 1986; Dansereau et 
al., 1975). 
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Applicability to ITS Development 


It is clearly desirable for an intelligent tutoring system to be able to offer learning 
strategies instruction to students. However, there are several practical issues which need to be 
addressed before it can be implemented. 

Hrst, taking time out for learning strategies instruction will cause scnne students to 
progress through the curriculum more slowly than others-at least initially. This is not necessarily 
a bad thing, especially if the students are really learning. However, within the context of a typical 
school environment where all students spend the same amount of time in the classroom, listen to 
lectures together, and take tests together, individualized instruction could be a management 
headache. Either supplementary instruction needs to be partitioned out in some way (e.g., so 
slower learners could receive learning strategies instructicHi after school, or during study periods), 
or the structure of a typical school day needs to be radically changed. 

Second, it may be difficult to target learning strategies instruction to individual learners. 
First the system must obtain the data it needs to decide whether or not such instruction is 
necessary. Then, if learning strategies instruction is best integrated with domain instruction, it will 
be necessary to figure out how to get the ITS to reorganize learning strategies infcnmation to 
adapt to particular situations. 


Summary 

This chapter has presented findings about specific teaching behaviors that could be 
implemented in intelligent tutoring systems. The next logical step is to integrate these behaviors 
into a single approach to effective instruction, and to represent it in a more computer-fiiendly 
form. 


ffl. SELECTION OF AN INSTRUCTIONAL APPROACH 

The purpose of this step was to select an instructional approach for initial in^lementation 
within an intelligent tutoring system. This chapter reviews some of the more influential 
instructional approaches including direct instruction, discovery learning, mastery learning, and 
cognitive apprenticeship. 

Before proceeding, we must identify what we mean by instructional approach. An 
instructional approach is; 

a theoretically guided set of instructional prescriptions or actions which 
facilitate^ the tranter of information from teacher to student. 

This working definition was selected because it illustrates an important characteristic of the 
instructional approach; namely, approaches have two distinct, but related components. The 
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theoretically guided portion of the definition identifies the first component. It refers to the set of 
assumptions made about the nature of teaching and learning. The set of instructional 
prescriptions or actions portion of the definition identifies the second component. It refers to the 
actual classroom activities which are required to instantiate the underlying assumptions. Simply, 
assumptions made about the nature of teaching and learning determine what activities are 
legitimate within a given approach. 

Suppose that we assume learning occurs passively. That is, we believe individuals learn 
by merely absorbing information from their environment Acceptance of this theory implies 
certain things. For example, telling students target content should be sufficient to insure learning. 
Lecturing, then, is a valid teaching action derived from the assumption that learning is a passive 
phenomena. Obviously, other teaching actions that involve telling students infeomation (e. g., 
give students a summary) are equally valid. The set of valid activities, taken together, comprise 
the activity structure of the approach. The activity structure is essentially the set of observable 
activities associated with a particular approach. Therefore, when most people speak of 
instructional approaches, they are referring to the activity structure or the observable 
characteristics of the approach. The assumptions which underiie these activities, however, are 
equally impxrrtant. As we shall see, faulty assumptions can lead to faulty instruction. 

Two problems complicate the task of defining the term instructional approach. First, most 
teachers do not adhere to a single approach; rather, they build eclectic riKxiels of instruction based 
upon their own experiences in the classroom. This is not necessarily a problem for us, however, 
because we are not dealing with classroom instruction. We are concerned with intelligent 
computer-based instruction. Second, differences between instructional approaches are often subtle 
and elusive. We address the second problem. Eventually, we will select an instructional approach 
suitable for intelligent tutoring system implementation. 

The overall goal of the Effective Teaching Behaviors project is to define and desenbe sets 
of effective teaching behaviors for intelligent tutoring systems. The instructional approach 
provides an ideal starting point because it essentially acts as a container which can be filled with 
effective teaching actions. By the container analogy, we mean to imply that instructional 
approaches have two properties. First, they have some general and unmalleable characteristics 
like teacher-direction of instruction. These characteristics define the approach and are analogous 
to the shape of the container. Second, there are many specific characteristics such as provide an 
advance organizer which are independent of instructional approaches. Many of these 
characteristics were identified in the first phase of this project. They are interchangeable in that 
we can add them in varying amounts to almost any approach. Therefore, they are analogous to 
the content of the container. Presently, we intend to select the best container for the effective 
teachii behaviors identified in the first phase of this project. In succeeding tasks, we will 
systematically integrate the effective teaching behaviors identified in the first phase of the project 
with the approach identified during this phase. Ultimately, the container and its contents will be 
used to devise a template of effective teaching behaviors suitable for implementation as the 
pedagogical component of intelligent tutoring systems. 


-‘52 






►.tJiimiTarroiE* 




IMIt 


Mir- 




Generally, there are four classes of approaches: 1) those based on objective and reception 
theories of learning (e.g., Ausubel, 1977; Rosenshine, 1979; Peterson, 1979), 2) those bas^ on 
inquiry theories of learning (e.g., Hermann, 1969; Nuthall & Snook, 1973), 3) those based on 
egalitarian and Socratic theories of learning (e.g.. Bloom, 1984; Slavin, 1987), and finally, 4) 
those based on cognitive and constructivistic doctrines (e.g., Palincsar & Brown, 1984; Cbllins, 
1988; (]TGV, 1992). Direct instruction, discovery learning, mastery learning, and cognitive 
apprenticeship fit these classes, respectively. Table 9 depicts some of the key similarities and 
differences which will be discussed in depth throughout the remainder of this document 


Table 9. Comparison of Direct Instruction, Cognitive Apprenticeship, and 
Discovery Learning. 


Variable 

Direct Instruction 

Cognitive 

Apprenticeship 

Discovery Learning 

Theory of learning 
implied by approach 

reception learning 

constructivism 

inquiry, experiential, 
(constructivism) 

Mode(s) of 
infonnation exchange 

telling of content 

telling, showing, 
doing 

doing 

Nature of the learning 

passive 

active, collaborative, 
experiential 

active, experiential 

Role of the instructor 

benevolent despot 

master, coach 

guide 

Role of the student 

sponge 

apprentice 

explorer 

Dominant teacher 
activities 

lecturing, questioning 
and testing students 

modeling, coaching 

observing student 
progress 

Dominant student 
responses 

listening, answering 
questions, completing 
homework 

watching, listening, 
doing, reflecting, 
articulating, exploring 

exploring 

Source(s) of feedback 

mostly from teacher 

teacher, task, self 

mostly from the task 

Ctontrol of pace of 
lesson 

teacher 

shared 

student 

Control of lesson 
content 

teacher 

teacher to shared 

student 

Degree of 
individualization 

teacher-centered, 
little individualization 

collaborative, highly 
individualized 

student-centered, highly 
individualized 
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For each approach we will: 

• discuss undeilying assuo^tions 

• discuss what these assun^tions entail in terms of instructional activides 

• identify in:q)oitant findings fitxn die professional literature 

• discuss implications for the use in intelligent tutoring 

Direct Instruction 

Direct instruction is perhaps the most common instructional approach. It is not an 
exaggeration to say that all American students have, at one time or another, been exposed to 
direct instruction. 

Assumptions 

Direct instruction is guided by a reception theory of learning. Ausubel states that, "in 
reception teaming the principal content of what is to be learned is presented to the learner in more 
or less final form" (AusubeC 1977). "More or less final form" means that the learner does not 
have to alter the information in order to use it Therefore, telling students the target content is 
sufficient to insure learning. 

ActiviOLStrucmrc 

Rosenshine has perhaps provided the best description of direct instruction (1979). He 
identifies the following attributes: 

• all activities are teacher-directed and monitored 

• a strong academic focus is maintained 

• content is highly structured and thoughtfully sequenced 

• goals are clearly and explicitly stated 

• the time allocated for instruction is sufficient and continuous 

• the coverage of content is extensive 

• the teacher structures student/teacher interactions 

• questions are at a low cognitive level to insure high rates of success 
(q)proximately 70%) 

• feedback is immediate and academically-oriented 

Thus, in direa instruction, the teacher controls aU facets of the instructional process 
including the instructional objectives, content, pace, the quality and quantity of student/teacher 
interactions, evaluation, and feedback. The primary mode of instruction is the lecture, although 
supplemaital media like movies, film, textbooks, and workbooks are also acceptable. Some 
researchers claim that up to 85% of the time in direct instruction is spent in lecture (Davis, 1983). 
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Ressarch 


Many direct instruction-related research studies have focused on the types of activities 
which positively correlate with achievement Brophy recognizes the following positive correlates 
of acactemic achievement (1986); 

• content covered 

• time allocated 

• academic-engaged time 

• clear, well-structured presentation of information 

• high rates of student success 

• regular and extensive feedback 

Rosenshine identifies extensive content coverage as an important characteristic of direct 
instruction (1979). Content coverage is roughly synonymous with the opportunity to learn; 
therefore, the importance of this variable is obvious. Simply, individuals with more opportunity to 
learn will tend to learn more. Measures of academic-engaged time, usually counts of on-task 
behavior, are often used as to indicate the amount of content covered. For example, the number 
of minutes per hour that a student is observed reading is a suitable measure of content coverage. 
The relationship between content covered and scholastic achievement is often assessed by means 
of correlations between counts of on-task behavior and achievement measures. Bloom (1976), 
for example, found correlations ranging from .40 to .52 between measures of student attention 
and scholastic achievement Stallings and Kaskowitz (1974) found a gmilar range of correlations 
between various measures of content covered and achievement 

Effective teacher management of the classroom is frequently cited as a correlate of 
achievement Rosenshine, for example, claims that teacher-directed classrooms are nxrre 
successful than non-teacher-directed classrooms (1979). A separate line of educational research, 
that of learner control, lends credibility to this claim. SpecificaUy, researchers have discovered 
that too much learner control of instructional events is detrimental to learning for certain types of 
information and student populations. Two explanations are advanced. Hrst, learners with too 
much control of instruction often either opt themselves out of instruction entirely or they choose 
the path of 'least resistance." Secondly, some learners may be ill-equipped to deal with the 
cogrtitive demands of simultaneously sequencing instructional events and learning novel material 
(Carrier, 1984; Steinberg, 1989). 

Other variables characteristic of direct instruction, such as low-level questions, controlled 
practice, and immediate and academically-oriented feedback are also associated with increases in 
achievement (Rosenshine, 1979). Rosenshine suggests that the activity structure of direct 
instruction promotes achievement relative to open or discovery-oriented approaches. Open 
approaches are antithetical to direct approaches, in that they are student-directed, rather than 
teacher-directed. Peterson confirmed Rosenshine's conclusion in a meta-analysis comparing direct 
and open approaches (1979). She found effect sizes which favored direct instruction for a variety 
of achievement outcomes. Effect sizes ranged from -.78 to .41 across several achievement 
measures (negative effect sizes favor direct instruction). The observed variability of effect sizes 
suggests two possibilities. First, direct instruction differentially affects, various achievement 
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measures. That is, the effects of direct instruction arc dependent on the measures selected. 
Second, some combination of statistical, experimental, and measurement errors inflated the 
variability of effect sizes. Despite the somewhat conflicting results, direct instruction seems to 
promote a variety of achievement outcomes. 

Peterson compared direct and open approaches in terms of affective outconres (1979). 
The results were somewhat mixed, but open approaches seemed to slightly promote outcomes 
like positive self-concept, school-attitudes and attitudes toward teacher slighdy more. Further, 
open approaches reduced anxiety and increased independence. 

Student characteristics seem to affect the relationships between direct or open instruction 
and achievement For example, Peterson cited a study by Ward & Barcho' in which low ability 
students performed equally well under direct and open approaches, but high ability students 
performed better under direct instruction (Peterson, 1979). This finding is in line with Ausubel's 
claim that learners must reach some developmental milestone before they are capable of resting 
the benefits of reception learning (Ausubel, 1977). Ausubel’s conclusion is controversiai, 
however. Grapko (1972), for example, found that the achievement of high ability students did not 
differ between direct and open instruction, but that low ability students performed better under 
direct instruction. While it is clear that student characteristics moderate the effectiveness of 
instructional approaches, it is not clear how. 

Implications for Intelligent Tutoring Systems 

Generally, direct instruction can be attacked on two points: 

• it encourages passivity which can lead to non-meaningful learning 

• achievement is not the only imponant educational outcome 

Reception learning is passive. Learners are assumed to be like sponges absorbing 
information from their environment (Schank & Jona, 1991). However, this assumption could be 
dangerous (Linn & Qancy, 1990). Passive learning facilitates idiosyncratic leaming~i.e., what is 
learned is mediated by what the learner already knows or believes. Large student groups, minimal 
student choice, and highly structured and limited student/teacher interactions create a situation in 
which student misconceptions can go undiagnosed. In addition, direct instruction encourages the 
learner to take what the teacher says as absolute fact because of the apparent omnipotence of the 
instructor. Rote memorization is encouraged and multiple perspectives or diverse viewpoints are 
discouraged (Schank & Jona, 1991). The result of this assumption is that learners often adopt 
expeditious, but ineffective, solution strategies which give the appearance of understanding but 
are actually based on some superficial aspect of the problem. Therefore, telling is not sufficient to 
insure learning. Simply put, learners often do not know enough about their own cognition to 
optimize learning (Winne, 1989). Telling does not insure that the student will attend to, 
assimilate, or store the information properly. 

Second, achievement is not the only desirable outcome of instruction (Peterson, 1979). 
Other outcomes, such as positive school attitudes and self-concept are also important. 
Unfortunately, advocates of direct instruction often base their arguments on the observed 
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achievement gains associated with its use. The instructional process should maximize as many 
positive outcomes as possible; direct instruction does not. 


These arguments do not mean that direct instruction cannot be an effective means of 
transmitting infonnation; it is. Rather, direct instruction may not always be the most effective 
means of instruction. Direct instruction is probably too limited for intelligent tutoring because it 
relies too heavily on only one mode of instruction: telling content. Intelligent tutoring systems 
afford educators with the means to instantiate much richer instructional approaches than direct 
instruction. 


Discovery Learning 

The discovery learning approach is largely student directed. In this sense, it is the 
theoretical opposite of direct instruedon. In fact, the pure forms of direct instruction and 
discovery learning are viewed as opposite poles of the instructional continuum (Hermaim, 1969). 
Instructional approaches probably do not exist in a pure form, however. Rather, they are mixed 
together, diluted, and altered. This is certainly true of discovery learning. Discovery is almost 
always tempered with some degree of guidance in practice. Generally, the diluted or mixed 
approach is called guided discovery. The following table offers a comparison of discovery 
learning and direct instruction. 


Table 10. Comparison of Discovery Learning and Direct Instruction. 


Discovery Learning 

Direct Instruction 

Students enjoy maximal control of content, 
pace, and sequence of instructional events. 

Students have few instructional choices. 


Interactions are highly structured and 
frequent. 

Feedback is infrequent It is likely to come 
from task performance, not the teacher. 

Feedback is more frequent and usually 
originates with the instructor. 

Materials ate not necessarily structured. 

Materials are structured. 

High-level, open-ended questions are used. 


Educational objectives are not necessarily 
clear. 

Objectives are explicit and clear. 
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Discovery approaches are founded on an inquiry theory of learning. Proponents of 
inquiry-based learning assert that learners find the process of discovery intrinsically challenging 
and rewarding; hence, outcomes like motivation, creativity, independence, and meaningful 
learning are facilitated (Nuthall & Snook, 1973; Peterson, 1979). 

Bruner (1961) claims that discovery learning is " a matter of rearranging or transfonning 
evidence in such a way that one is enabled to go beyond the evidence so reassembled to additional 
new insights”. Therefore, it seems clear that the implicit objective of discovery learning is to place 
students in an inquiry mode which requires the use of inductive processes. 


The activity structure of the discovery approach is ambiguous because of the complex 
nature of inquiry and the pervasive disagreement about how to (^rationalize discovery. One 
defmition, for example, says that the discovery learning approach is two-stagecL In the first stage, 
students are asked to solve problems or discuss relevant examples from experience. In the second 
stage, the problem and/or example is altered. The student must conconutantly alter the solution 
algorithm or strategy in order to successfully cope with the problem. During this stage, the 
student discovers new principles or methods which facilitate navigation of the problem space 
(NuthaU & Snook, 1973). 

This is only one concepmalization of the discovery approach, however. Davis (1983) 
claims that discovery learning is really a process which involves a sequence of four steps. First, 
the learner must sense a problem or discrepancy in some set of knowledge. Presumably, in a 
guided version of discovery learning, the teacher would create the problem or discrepancy. 
Second, the learner must define the problem and formulate a solution plan. Next, the learner must 
search for relevant information, hypothesize about the problem, and occasionally backtrack to 
previous parts of the sequence. Hnally, the learner must resolve the problem or eliminate the 
feelings of "disequilibrium." 

Defining the activii) fructure of discovery learning is a difficult task because of the 
disagreement among discovery learning proponents. It has been suggested that the only common 
theme present in the assortment of definitions of the discovery approach is that it is not "telling" 
(Nuthall & Snook, 1973). The role of the student is clear. He or she must work through some 
problem inductively arriving at some new concept, principle, or theory. The role of teacher, 
however, is less clear. 

Based upon these various definitions, we suggest that the role of the teacher in guided 
discovery is: 1) to provide some initial problem or topic of inquiry and 2) to provide feedback 
when necessary. AH other phases of the instructional process seem to be the responsibility of the 
student. 






Research 


Hennann (1969) offers the foUowiiig summary of discovery learning findings: 

• Rule-exaiiq)le learning (a technique often used in direct instruction) pixunotes 
retention relative to discovery. In rule-example learning, a rule is presented 
followed by a specific exati:q)les. 

• Discovery approaches promote transfer relative to direct instruction. That is. a 
student who has discovered some principle, concept, or thetny on his own is more 
able to apply it some other domain of knowledge. 

• As the difficulty of the transfer task increases, the efficacy of discovery learning 
also increases. In other words, the more dissimilar the transfer task is from the 
learned task, the more effective discovery learning becomes. 

• Discovery learning may be more effective when the learner's subject knowledge is 
limited. This suggests that novices should be taught with discovery methods 
although Husic, Linn. & Sloane (1989) suggest otherwise. 

• Discovery learning is more effective when the material is academic. 

• The discovery method is more effective with low ability than with high ability 
populations. 

• After material has been learned by discovery, immediate verbalization or further 
learning adversely affects the original learning. Therefore, it is probably not a 
good idea to give learners a sununary or post-organizer following discovery 
learning. 

• A "reasonable" degree of guidance is better than no guidance. Hermann does not 
explain what he means by a "reasonable" degree of guidance, but logically some 
instruction and feedback will improve performance. 

In an interesting study, Husic, Linn, & Sloane (1989) found that teachers with 
introductory programming classes tended to use direct instruction while teachers with advanced 
classes were more likely to use discovery methods. These choices seemed to be warranted on the 
basis of achievement There were significant positive correlations between instructional approach 
and student performance. For example, in the introductory classes "identifies important 
infexmation" correlated .71 with achievement while "encourages independent inquiry" correlated 
.50 with achievement in the advanced classes. As expected, "encourages independent inquiry" 
correlated -.70 with achievement in the introductory classes. 

As mentioned in the previous section, Peterson found that open or discovoy approaches 
slighdy facilitated some affective outcomes like positive attitude, self-concept, indq>endence, and 
creativity (Peterson, 1979). Hermann reported that discovery methods promoted transfer (1969) 
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and Husic ct al. found that discovery learning was effective for advanced students. Thus, some of 
the claims of discovery learning proponents are supported. 


Implications for Intelligent Tutoring Systems 

Although discovery learning methods have a place in instruction, several factors make 
total reliance on discovery impractical, if not unfeasible. First, conceptual definitions of discovery 
learning are ambiguous. Therefore, it is difficult to accurately instantiate the true meaning of 
discovery. Second, discovery learning is inefficient because the amount of content covered 
relative to time expended is low. Learners spend a lot of time while covering only s mall amounts 
of material. Third, the nature of discovery insures higher rates of student failure and the 
possibility of concomitant decreases in motivation, attitudes, and self-concept. Rnally, discovery 
learning promotes a disparity- between the cognitive experiences of the teacher and student. 
Therefore, it is difficult for the teacher to provide adequate feedback (Nuthall & Snook, 1973). 

We suggest that discovery techniques do not provide adequate or complete instructional 
capabilities for intelligent tutoring because of vague instructional characteristics, inefficiency of 
learning in terms of amount of material covered relative to time invested, and problems associated 
with the lack of guidance. Discovery can be extremely useful as an instructional supplement, but 
it does not provide the ITS developer a full palette of options. 


Mastery Learning 

Mastery learning approaches emphasize student mastery of content, not the amount of 
content covered. There are several mastery ' ariants which, at first glance, appear to be entirely 
different instructional approaches. They do, however, share a common philosophy. 

A.s.sumptions 

Mastery learning approaches are guided by an egalitarian philosophy of education. That 
is, all individuals are given equal opportunity to excel in an individualized environment This is 
accomplished by holding student achievement constant while time is allowed to vary. Normally, 
significant positive correlations are observed between intelligence and scholastic achievemenL 
Thus, individuals with lower intelligence are placed at a disadvantage. Mastery learning 
approaches circumvent this problem by giving ail students enough time to master the content 
(Keim-Abbott & Abbott, 1977). 

Activity Structure 

In traditional classroom-based learning, there is a finite time frame imposed on upon each 
lesson. Generally, smdents arc tested at the end of the each lesson. This means that ail students 
have the same amount of time to learn the material. If students have not mastered the content by 
the time of the test, they do not get another chance. In contrast, mastery learning holds 
achievement, not time, constant. Students must meet or exceed some mastery criterion in order 
to proceed, regardless of the time required. This allows more students the opportunity to master 
the content. 
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There are at least two instantiations of mastery learning; the Keller Personalized System 
of Instruction and Bloom's mastery learning approach. The Keller Personalized System of 
Instruction is essentially a discovery-oriented mastery approach. Students are given a study guide 
which they use to guide themselves through a lesson. Students can anend a regularly scheduled 
class, work on their own, or smqsly take the test without studying the material. Occasionally, the 
instructor lectures students on important issues or concepts. Students must pass a test in order to 
proceed to the next lesson. If a student fails the test, there is no penalty, but they must retake the 
test. (Kulik, Xuhk, & Cohen, 1979). 

Bloom's mastery learning is somewhat like direct instruction. The key difference lies in 
the fact that Bloom's system provides for individualization of instruction. In Bloom's (1984) 
system: 


Students learn the subject matter in a class with about 30 students per teacher. 

The instruction is the same as in the conventional class... Formative tests...are 

given for feedback followed by corrective procedures and parallel formative tests 

to determine the extent to which the students have mastered the subject matter. 

Bloom's notion of mastery learning capitalizes on some of the correlates of academic achievement 
(e.g., academic focus, use of teacher-directed and monitored activities) mentioned previously. 

Whatever the form, the key features of mastery learning are: 

• A mastery criterion. The teacher sets a minimum level of success which must be 
met 

• Small segments of content Instruction is divided into small segments. 

• Student control of pace. The student is allowed as much time as necessary to 
complete a lesson. 

• Frequent assessment Tests follow each lesson. They determine whether the 
student moves to next lesson or receives remediation. 

• A system of corrective feedback and remediation which capitalizes on the benefits 
of individualized instruction. Feedback and remediation are tailored to individual 
needs of each student. If a student is successful in one area and ^successful in 
another, he or she receives help only in the problem area. 

Research 

Bloom's 1984 paper identifies the two-sigma problem. Essentially, the 2-sigma problem 
refers to the fact that one-on-one tutoring results in achievement gains of two standard deviations 
over conventional methods of instruction. Thus, the problem is how to attain these levels of 
achievement without using one-on-one tutoring. According to Bloom, mastery learning provides 
the best answer because of the individualized nature of the instruction. 

Bloom (1984) claims that mastery learning allows 80% of the students to attain a level 
typically reached by only 20% under traditional methods. Therefore, more students have the 
opportunity to excel. A meta-analysis by Slavin (1987) reported smaller effect sizes than those 
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found by Bloom; nonetheless, they also favored mastery learning. However, Slavin noted that 
when time was controlled, positive effect sizes nearly disappeared. Additionally, Slavin found that 
the achievement gains produced by mastery learning were strongly nuxierated by the tyjre of test 
used. Experimenter-made tests showed larger effect sizes than standardized achievement tests. 

A meta-analysis of Keller's Personalized System of Instruction (PSI) also showed positive 
effect sizes (Kulik, Kulik, & Cohen, 1979). Analysis of variance revealed that Keller’s PSI is 
equally effective for high and low ability students. Additionally, Keller’s PSI seemed to be more 
effective in certain domains, such as the social sciences; and essay test performance was enhanced 
relative to objective test performance. Mastery learning approaches seem to enhance student 
achievement and have obvious implications for self-concept, motivation, and school attitudes. 

Implications for Intelligent Tutoring Systems 

The literature clearly indicates that mastery learning improves achievement over 
conventional methods because of the individualized nature of instruction (Bloom, 1984). There 
are several problems, however. First, the facilitative effects of mastery learning tend to diminish 
or disappear when the effects of time are controlled as is demonstrated by Slavin (1987). Many 
mastery learning studies do not control for the effects of time. Control group subjects get only X 
amount of time while mastery learning subjects receive all the time they need to master the 
content. It is no wonder that they achieve at higher levels. Second, a decision must be made 
about what to do with learners who finish before their classmates. Should they go on, wait for 
their classmates to catch up, or review the same material in more detail? Finally, some attribute 
the positive effect sizes favoring mastery learning to the fact that teachers key on testable 
concepts. Slavin’s study showed that this is probably true. These issues indirectly affect the 
decision to use mastery learning approaches in intelligent tutoring. If, for example, the 
achievement gains associated with mastery learning are the result of time effects, then we may be 
less eager to use the approach in intelligent tutoring. 

The mastery learning philosophy and its implications for individualized instruction are 
ideally suited to intelligent tutoring systems. The computer allows each student to progress at his 
or her own pace without raising the problem of what to do with quick-finishers. We suggest 
applying the mastery learning philosophy to intelligent tutoring system design because it is fairer 
to lower ability students, it allows more students to master the material, and it facilitates more 
individualized instruction. In fact, many intelligent tutoring projects already utilize the notion of 
mastery. 


Cognitive Apprenticeship 

Cognitive apprenticeship is an approach characterized by the showing, telling, and doing 
of authentic tasks in authentic contexts. It is designed to reduce the problem of inert or useless 
knowledge. The fundamental goal of education is to develop and foster sets of working 
knowledge and strategies which can be applied to real world problems. Unfortunately, all too 
often the more immediate goal of short-term retention or cramming predominates. As a result, 
students seem unable to apply facts or concepts learned within an academic setting to problems 
outside of that setting. Cognitive apprenticeship was designed to promote transfer of knowledge 
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to real world settings, by eliminating the artificial separation of knowing and doing that 
characterizes conventional instructional methods (Brown, (jollins &. Duguid, 1989; Collins, 
1988). 

Assumpiions 

cognitive apprenticeship derives from several ideas about the nature of cognition, most 
notably, constructivism. Merrill (1991) identifies the following tenets of constructivism; 

• Learning is constructed from experience. 

• There is no shared reality. 

• Learning is an active process. 

• Learning is coUaborative-that is, meaning is derived from multiple perspectives. 

• Learning is situated. 

• Testing should be an integral component of the learning task 

In the following section, we will examine the tenets of constructivism in detail. This does 
not mean that advocates of cognitive apprenticeship are pure Lbntructivists; however, many of 
the assumptions of constructivism are intricately tied to the activity structure of cognitive 
apprenticeship. 

The notion that knowledge is constructed from experience implies that education should 
be experientiaL This assun^tion directly contradicts objectivism. Objectivists are opposed to the 
idea that knowledge is constructed from experiences with the world. Experience, they say, does 
not play a role in the structuring of knowledge because the properties, entities, and relations in the 
worid are already cotiqjletely and correctly structured (Duffy & Jonassen, 1991). Traditional 
methods of instruction such as direct instruction are closely aligned with objectivism. Proponents 
of cognitive apprenticeship, on the other hand, believe that knowledge is constructed from 
experience. However, this belief does not preclude telling students information. Rather, 
cognitive ^prenticeship allows for both experiential learning and telling. 

The second assumption, that there is no shared reality, is more controversial. On one 
level, this assertion is quite true. That is, no two people in the world share exactly the same 
experiences. Therefore, it seems reasonable to conclude that each individual has deq>ly personal 
perceptions of what reality is and isn't On another level, however, this assumption is totally 
absurd. Obviously, the majority of our knowledge of the world is not idiosyncratic but shared 
(Merrill, 1991). Some knowledge, in fact, must be common to all people. For example, in our 
culture we all know that red means stop and green means go. What would happen if everyone 
had their own idea of the meaning of a stoplight? Without question, operating a motor vehicle 
would become inordinately dangerous. However, although this assumption is wrong in strictest 
sense, the main point of the Constructivists is that people do not always see things in the same 
way. 

The third assumption, that learning is an active process, is generally supported by the 
cognitive literature. We do not merely absorb infenmation; rather, we construct it, based upon 
what we already know and believe (linn & Qancy, 1990, Schank & Jona, 1991; Smith, 1988). 






Thus, it seems only natural to use instructional approaches which capitalize on our constructive 
tendencies. 

The fourth assumption, that learning is a collaborative process, stands in opposition to 
direct instruction. Collaboration and multiple perspectives are important educational tools. In 
direct instruction, there is only one authority, the instructor, and usually only one viewpoint 
Proponents of constructivism, on the other hand, hold that meaningful learning occurs when 
learners examine multiple perspectives. Collaboration with others encourages the examination of 
multiple perspectives, which in turn serves to flesh out knowledge and promote social goals. As 
was observed in the discussion about direct instruction, exposure to relevant academic stimuli is 
positively correlated with achievement Exposure to multiple perspectives singly represents 
exposure to more complex and diverse academic stimuli. It follows, then, that collaboration and 
multiple perspectives should improve the quality of information available to the student 

The fifth assumption, that learning is situated, is quite important. The implication is that 
all learning should occur in the context or sphere of its use in the real world (Collins, 1988; 
Wilson & Cole, 1992). Traditional instruction ignores this maxim. Learning in conventional 
classrooms separates knowledge from its uses in the real world. Students learning to solve 
algebra word problems, for example, are often given unrealistic and uninteresting examples, which 
serve purely pedagogical purposes. Students come to associate algebra with academic settings 
and irrelevant word problems. As a result, they fail to see the practical uses of algebra. It should 
be noted, however, that other researchers claim that the notion of situated cognition is extreme 
(Sandberg & Wielinga, 1992). Meaningful learning can occur in the absence of strict situation. 
Additionally, sometimes situated cognition can lead to solution strategies which arc situation- 
specific. Nonetheless, the preponderance of the evidence suggests that situated learning results in 
the development of effective solution strategies (Qanccy, 1992). Authentic learning tasks 
capitalize on the benefits of situated cognition. Learners discover solution strategies in the 
context of their use. This serves to structure knowledge relative to its use in the real world. 
Therefore, situated learning reduces the problem of inert knowledge (Collins, 1988). 

The final assumption of constructivism states that tests of learning should be integrated 
with the learning task itself. This means that demonstrations of competence should occur as a by¬ 
product of the interaction of ine learner with real tasks and materials during the learning process. 
Traditionally, tests are separated from the rest of the instructional process. This is a dangerous 
situation, in the contructivist view, because it further serves to remove knowledge from the sphere 
of its use. 

Activ ity Str»gm.rg 

Cognitive apprenticeship is characterized by the showing, telling, and doing of authentic 
tasks in authentic contexts. It represents a return to the resource-intensive mode of instruction 
characteristic of the traditional trade apprenticeships. In terms of the variety of activities it calls 
for, it is the richest of the instructional approaches reviewed so far. The following sections will 
describe the activity structure of cognitive apprenticeship in greater detail. 

Content. Generally, four types of knowledge are taught in cognitive apprenticeship: 1) 
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domain knowledge, including conceptual, factual, and procedural infOTmarion; 2) heuristic 
strategies or "tricks of the trade"; 3) control strategies; and 4) learning strategies (Collins, Brown, 
& Newman, 1989; Cbllins, 1988; Wilson & Cole, 1992). In contrast, conventional methods cmly 
attend to domain knowledge. Strategy instruction is wholly ignored despite the fact that many 
researchers have shown that cognitive strategy instruction facilitates student achievement ( e.g., 
Palincsar & Brown, 1984). Simply, students must be instructed in tacit, heuristic knowledge as 
well as textbook knowledge (Collins, 1988). 

Sequence. Content should be sequenced fiom simple to complex, with increasing diversity 
of information, and from global to local skills (Wilson & Ctole, 1992). Simple to complex 
sequencing means teaching easier, lower-level material before harder, higher-level material. 
Increasing diversity of material means that as the lesson progresses, more examples and practice 
contexts are employed. Fmally, the global/local skills distinction means that it is the teacher's job 
to help the students acquire a general mental model of the problem early in the instruction. Only 
later does the teacher instruct specific skills. For example, suppose that we arc teaching students 
to add. Hrst, we would teach students single-digit addition. We would be certain to emphasize 
uses for addition skills in the real world. Gradually, we would begin to include multiple-digit 
problems. We would begin to provide examples of how to apply addition knowledge to specific 
tasks like word problems. Ultimately, we would provide smdents with simated learning 
environments like running a imaginary store or keeping records for a imaginary bank. 

Nature of the learning task. Content must be taught or situated in the context or sphere of 
its use to avoid the problem of inen knowledge (Collins, 1988; Lajoie & Lesgold, 1989; Wilson & 
Cole, 1992). Knowledge is traditionally taught in abstract ways which promote the use of 
superficial and largely ineffective strategies; that is, strategies which facilitate shon-term retention 
of material at a highly superficial level (Wilson & Ctolc, 1992). Students, for example, may merely 
key in on some superficial aspect of a word problem which allows them to answer the question 
correctly without fully understanding the concepts involved. 

C2ollins suggests the use of multiple contexts in instruction because they foster both 
general and specific knowledge (1988). Multiple contexts refer to different scenarios which 
require the use of the same skill. The use of authentic, multiple contexts provides the learner 
with specific examples of how know ledge is used in the real world. Providing several example'’ 
allows the student to generalize across those examples and contexts. Thus, knowledge becomes 
both specific and general. 

Supposedly, situated learning has the following benefits: 1) students leam the conditions 
for applying the knowledge, 2) invention is fostered, 3) students see the implications of the 
knowledge, and 4) authentic contexts structure knowledge relative to the sphere of its use 
(Collins, 1988). In short, situated learning reduces inert knowledge. 

Instruction. As mentioned, cognitive apprenticeship has showing, telling, and doing 
activity structures. The teacher shows the student how something works or how to do 
something, tells why it is that way, and then the student docs it 

Generally, some process or action is modeled. The teacher shows how the process or 
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action unfolds and'simultaneously tells the reasons why it happens that way. It is important that 
the teacher model and explain, or show and tell, simultaneously because the student needs access 
to explanations as he or she observes the details (Collins, 1988; Wilson &. Cole, 1992). Collins 
identifies two forms of modeling: modeling of processes as they occur in the world and modeling 
of expert performance (1988). Modeling physical processes is relatively straightforward. 
Usually, this can be accomplished with a simulation. The simulation abstractly illustrates how the 
process unfolds while simultaneously providing reasons for observed changes in the simulation 
variables. Modeling expert performance is somewhat more difficult The nxxleling should 
include false starts, dead ends, and backup strategies in addition to general solution algorithms 
(Wilson & Cole, 1992). This maximizes the authenticity of the process. In other words, the 
novice receives information about how to handle erroneous solutions, not just correct ones. 

Supposedly, the benefits of modeling include: 1) seeing expert solutions to problems set 
by the studjnt, 2) integrating what happens and why, and 3) making parts of a process not 
normally seen visible (Collins, 1988). Essentially, the modeiing/expiaining phase serves to 
structure future instruction. 

The next and most important component of the instructional process is coaching. The 
student receives the bulk of the necessary information during coaching. A scc^oldinglfading 
paradigm is employed during coaching to insure that the student actively attacks a problem and 
does not merely wait to be told the correct answer. Basically, scaffolding and fading refer to 
inter-related processes of providing the student with information as it is needed. The teacher 
observes the student as he or she tries to solve some problem and provides minimal quantities of 
information when the student is unable to proceed (Wilson & Cole, 1992). 

The academic scaffold is analogous to its real world counterpart. Both are temporary and 
movable supports which allow individuals to complete a task that would otherwise be impossible. 
The notion of the academic scaffold is based on the work of Russian cognitive scientist Vygotsky. 
Vygotsky (1978) hypothesized about the zone of proximal development. The opposite sides of 
the zone represent what the student is capable of doing independently and what the student is 
capable of doing only under the guidance of the teacher. The role of the teacher is to provide 
just enough support so that the student begins to navigate the zone. Eventually, as the student 
demonstrates proficiency, the amount of help is gradually reduced or faded until the student is 
able to independently solve the problem. The type and amount of knowledge required is at the 
discretion of the teacher. 

The benefits of coaching include: 1) coaching provides help directed at real difficulties, 2) 
coaching provides help at critical times, 3) coaching provides as much help is needed to 
accomplish the task, and 4) coaching gives the student new perspectives (Collins, 1988). 

The coach must elicit a series of authentic performances from the student in order to 
diagnose student knowledge and determine the effectiveness of various interventions. Generally, 
performance is elicited by having the student work through some problem, but there are three 
specialized forms of performance which are especially important to cognitive apprenticeship. 
These are articulation, reflection, and exploration. 
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In articulation, students are prompted to think about what they are doing and to provide 
reasons for their decisions and choice of strategies. Supposedly, this aids in making tacit 
knowledge explicit (Collins, 1988). Therefore, knowledge about specific aspects of performance 
which is normally implicit becomes available for other tasks. Ardculation allows students to 
compare solution strategies across situations and develop general solution strategies. 

In r^ection, students analyze their performance and compare it to the performance of 
others. Performance becomes a subject of study; therefore, students become more aware of their 
strengths and weaknesses (Collins, 1988). Reflection and articulation arc similar, in that both 
require the student to think about actions and processes which are normally automatic. 
Presumably, this facilitates metacognirive development 

In exploration, students try out alternative hypotheses and strategies in some simulated 
domain. This activity draws from discovery learning . Discovery learning is less efficient than 
direct instruction for simple content acquisition, but when the instructional goal is independent 
problem solving, exploration becomes more important (Wilson & C!olc, 1992). Therefore, 
exploration helps the learner to go beyond the simple and superficial thinking characteristic of 
approaches which emphasize merely telling students the target content 

Generally, cogrutive apprenticeship can be viewed as an approach which uses both 
showing and telling and requires doing. In this sense, it has the most complex activity structure of 
any of the approaches addressed here. 

Research 

Reciprocal teaching has been identified as a successful form of apprenticeship (Cbllins, 
Brown, & Newman, 1989; Wilson & Cole, 1992). Briefly, in reciprocal teaching, die teacher and 
students take turns leading a dialogue about the important features of some text. During 
discussions about the text, the students are taught four strategics which are designed to foster 
student comprehension and comprehension monitoring (Palincsar & Brown, 1984). Reciprocal 
teaching has the three general characteristics of the cognitive apprenticeship activity structure: 
showing, telling, and doing. Additionally, some of the more specific characteristics of cognitive 
apprenticeship are preserved. Collaboration, exposure to multiple perspectives, situated learning, 
and scaffolding are all integral parts of reciprocal teaching. 

Palincsar & Brown have shown that reciprocal teaching results in superior achievement 
and retention over conventional methods for seventh grade problem readers (1984). In a meta¬ 
analysis, Rosenshine and Meister (1991) confirmed this conclusion. They reported a median 
effect size of .52 indicating that reciprocal teaching facilitates the comprehension of text over 
conventional methods. They found that reciprocal teaching was more effective when cognitive 
strategy instruction was made explicit, although it did not matter how many strategies were 
taught. 


Atkinson provided an ethnographic demonstration of the effects of apprenticeship training 
in ar academic setting (1989). She reported that her apprenticeship reading students; 1) liked 
reading nnore; 2) liked to share reading more with friends, parents, and teachers; and 3) 
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demonstrated substantially higher levels of motivation than other students. In another study, 
English as a Second Language students were taught English and math using cither an 
apprenticeship approach or conventional methods. Scaffolding was provided by giving students 
help in their native language. The authors found that one year after its introduction, the 
apprenticeship program helped these students to improve significantly over control students in 
English and math. They concluded that the use of modeling, scaffolding, evaluation, and 
collaboration were responsible (Thornburg & Karp, 1992). 

Cognitive apprenticeship has also been tested in technical domains. SHERLOCK is a 
computer-coached practice environment for F-15 repair technicians which utilizes an 
apprenticeship approach. Lajoie & Lesgold (1989) repon that subjects who spent 25 hours on 
SHERLOCK were as competent as controls who had spent four years on-the-job. Additionally, 
SHERLOCK subjects solved significantly more problems than their control counterparts, showed 
more expert-like problem-solving steps, and made fe wer incorrect or bad moves in problem¬ 
solving than their control counterpans in on-the-job situations. 

A similar tutor designed to improve electronics troubleshooting skills using computerized 
case-based simulations reported similar results (Johnson, 1992). The mtor resulted in significant 
gains in troubleshooting ability of college electronics students. Subjects with only 5 hours of 
exposure to the tutor demonstrated a 78% improvement over the control group. Johnson also 
reported that the experimental subjects were more determined to locate problems and displayed 
greater confidence in their diagnoses despite the fact that both groups showed equal domain 
knowledge as measured by a paper-and-pencil test. He concluded that the apprenticeship tutor 
improved the cognitive and metacognitive processes of the experimental group (Johnson, 1992). 


Cognitive apprenticeship is a relatively new approach to instnxtion. Therefore, there have 
not been many empirical studies of its effectiveness. Nonetheless, from those that exist, it seems 
that cognitive apprenticeship is an effective approach in academic and technical settings. 

Implications for Intelligent Tutoring Systems 

The main criticism of cognitive apprenticeship is that it is resource intensive. That is, it 
requires a lot of time, money, effort, and materials (Atkinson, 1989). Conversely, this can be 
viewed as a strength because students receive more detailed and diverse instruction. The 
computer provides an excellent means of instantiating the approach because the computer can; 

• provide a vast array of materials and media choices 

• modei processes which are not normally visible 

• present materials simultaneously using different media 

• constantly monitor student performance 

• provide individualized coaching and feedback 

CogTutive apprenticeship is by far the most comprehensive model of instruction. It 
includes the important elements of all of the other approaches, such as telling content and doing 
real world tasks, which the coach can invoke when deemed necessary. For example, research 
indicates that introductory students perform better under direct instruction, while more advanced 











students benefit from some foim of discovery (Husic, Linn, & Sloane, 1989). A cognitive 
apprenticeship-based tutor could simultaneously provide both of these environments. The novices 
could receive marc structured coaching and low-level performance elicitation; and the higher-level 
students could be in an exploratory mode in which they could formulate hypotheses and discover 
rules. Further, the mastery spirit is maintained. Coaching individualizes instruction. 




icndations 


In conclusion, it seems logical to use cognitive apprenticeship as our instructional 
approach, for the following reasons; 1) the inherent limitations of the other approaches, 2) its use 
does not preclude using techniques characteristic of other approaches, and 3) it is compatible 
with the goals of intelligent tutoring systems. 

Direct instruction and discovery learning have limitations which constrain their use in 
intelligent tutoring. Direct instruction is an efficient means of transmitting information from 
teacher to student, but it can encourage passive, idiosyncratic learning. This can result in student 
misconceptions which are hard to diagnosis and fix. Discovery learning encourages active 
learning, but it is inefficient Students spend a lot of time learning a little information. In addition, 
rates of student failure are increased, which has negative implications for student motivation and 
attitudes. Table 9 illustrates some specific advantages of cognitive apprenticeship. For example, 
for the variable mode(s) of information exchange, direct instruction and discovery learning utilize 
only one mode, whereas cognitive apprenticeship utilizes three. Therefore, the nature of the 
learning in cognitive tq)prenticeship is more detailed and diverse. 

Cognitive apprenticeship is an eclectic approach which does not preclude the use of 
specific techniques from the other approaches. Therefore, cognitive apprenticeship not only 
avoids the problems associated with other approaches, but also capitalizes on their strengths. For 
example, direct instruction primarily utilizes telling; the teacher tells students important 
information. Discovery learning primarily utilizes doing; students are required to do something. 
Cognitive apprenticeship incorporates both telling and doing, as well as showing. Therefore, in a 
sense, it subsumes the other approaches. 

Hnally, cognitive apprenticeship is compatible with an important goal of intelligent 
tutoring systems-individutdization of instruction. As illustrated in Table 9, individualized 
instructiem is highly emphasized in cognitive apprenticeship. One of its key components is 
coaching which, like tutoring, provides help: 1) d^ted at real difficulties, 2) at critical times, and 
3) only as needed. Therefore, students are taught according to their individual needs. 

During the next phase of this project, we will demonstrate how to integrate the cognitive 
apprenticeship approach with our knowledge on effective teaching behaviors. The demonstration 
will take two forms. First, diagrammatic representations of the knowledge will be presented. 
Second, a system of production rules will be developed. The rule-based representation of this 
knowledge will be directly transferable to many intelligent tutoring systems. 
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rV. SELECTION OF KNOWLEDGE REPRESENTATION FORMAT 


The selection of a knowledge representation fomiat is an important decision, with 
significant implications for an intelligent tutoring system’s capabilities and overall efficiency. As 
used here, the term knowledge representation format includes: 1) the techniques used to 
represent the knowledge; and 2) the methodology for organizing the knowledge in the system 

Review of Kno wledge Representation Techniques 

A knowLJge representation technique is a means for expressing knowledge so that a 
computer can use it. There are a number of different techniques which can be used, and 
combinations are also possible. However, not all of these techitiques are necessarily appropriate 
for representing tutoring knowledge. In selecting wliat technique(s) to use, it is important to 
consider the type of knowledge to be represented and the way that knowledge will be used. 
Knowledge about mathematical theorem-proving, for example, is represented differently from 
knowledge about disease diagnosis. A brief description of four of the most ftequently mentioned 
techniques follows. 

• Production rules - Knowledge is represented by IF-THEN conditional statements. 
The IF, or condition part, states the conditions which must be present for the rule 
to be applicable; the THEN, or action part, specifies the action that will take place 
if the condition is true. The computer reasons by processing rules until it finds one 
it can use. Production mles are particularly useful when the knowledge to be 
represented is easily expressed in rule form; they are frequently used for design 
and/or diagnosis problems in engineering and medicine. Example systems include 
MYCIN, which aids in diagnosis and selection of therapy for patients with 
meningitis (Barr & Feigenbaum, 1982); and the VLSI Design Automation 
Assistant, which helps engineers to design microprocessors (Brachman, 1986). 

• Predicate logic - Knowledge is represented using predicate logic (e.g., at(Joe, 
office) or event(landed(Columbus),I492)). The advantage of using this technique 
is that standard theorem-proving procedures for this type of reasoning already 
exist. The computer sinqrly keeps applying rules of deduction until it reaches the 
goal state. This technique is most useful when the knowledge to be represented 
can be expressed in mathematically precise terms. It is most commonly used for 
problem solving and theorem-proving. Examples include FOL, which checks 
proofs stated in first-order logic; and STRIPS, which helps a robot with the 
planning problems involved in moving objects and navigating in a cluttered 
environment (Barr & Feigenbaum, 1982). Two disadvantages of predicate logic 
are: 1) it can be very slow (Brachman, 1986); and 2) representing certain kinds of 
information, such as amount of certainty and heuristic information, is difficult 
(Rich, 1983). 
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• Semantic networks - Knowledge is represented as networks of objects, concepts, 
or situations which are linked according to their relationships to each other (e.g., 
is-a, owned-by, or is-a-property of). The computer reasons by checking the 
relationships between different nodes in the network. This technique is useful 
when knowledge can be broken down into small, interconnected units; it is often 
used for natural language understanding programs (Barr & Feigenbaum, 1982; 
Brachman, 1986). One of the earliest intelligent tutoring systems, SCHOLAR, 
used a semantic network to represent and to teach knowledge about South 
American geography (Carbonell, 1979). 

• Frames and Scripts - Knowledge is contained in groups of related ideas (like 
schemata). A restaurant frame, for example, would contain knowledge about 
things commonly found in restaurants (such as waiters, tables, food, etc.) and 
expectations for different types of restaurants. A script would contain 
expectations about the sequence of events that usually occurs when one enters a 
restaurant. Frames and scripts are like semantic networks in that they can be 
linked according to relationships; but usually represent more complex forms of 
knowledge. Frames and scripts are useful for applications requiring recognition or 
prediction. Examples Lnclude (TENTAUR, a hybrid system for medical diagnosis, 
which uses frames to organize production rules into groups relevant to particular 
diseases (Brachman, 1986); and NVHY, an intelligent tutoring system, which uses 
scripts to represent knowledge about the causes of rain (Stevens, (Zollins, and 
Goldin, 1979). 

Most intelligent tutoring systems have used production rules to represent teaching 
knowledge (Rickel, 1989). Exceptions include Carbonell’s SCTHOLAR (1979), and Stevens, 
Collins & Goldin's WHY systems, which were described above. Both of these systems are fairly 
old, however, and it appears that the knowledge representation techniques they used were 
selected for their fit with the domain knowledge, rather than for their ability to represent 
pedagogical knowledge. In fact, Stevens, Cbllins, and Goldin point out that WHY is quite 
limited, in terms of its abilities to diagnose student errors and to explain physical processes (both 
important pedagogical functions), because of its use of a script representation. 

Given that the ultimate goal of this effort is to develop one or more generic teaching 
knowledge templates that can be imported into a variety of tutoring systems, a rule-based 
representation seems most appropriate. This is because: 1) rules are a widely-used knowledge 
representation technique which reduces the potential for incompatibility; 2) rules are relatively 
easy to implement in most programming languages, and programmers are familiar with their use; 
3) rules are easy to add, delete, and modify, which will be useful if the template needs to be 
updated or tailored to the needs of a specific tutoring application; and 4) the knowledge about 
effective teaching behaviors identified up to this point seems to lend itself naturally to being 
expressed in rule form. 
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A system organization methodology is a scheme for structuring and utilizing the 
knowledge within a system in order to facilitate problem solving (i.e., in order to reach a desired 
goal state from the current state). Given that production rules will be utilized, there are several 
different methods for organizing a system (Rich. 1982; Simons, 1984). The most promising (for 
our purposes) include; 

• List of rules. In this scheme, the system searches through all the rules until it finds 
a rule it can use. If the rules are few in number, this is an acceptable approach. 
However. 1) as the number of rules increases, it becomes difficult to add new rules 
which do not conflict with existing ones; and 2) it is inefficient to consider all rules 
at each step of the problem solving process—e.g., sometimes particular clusters of 
rules apply together, and only when they finish should other rules be considered 
(Rich, 1982). 

• Model-hosed. Rules are used to establish and, over time, correct a model of the 
relevant world (Simons, 1984). This approach is particularly useful for 
representing the current state of affairs, or the way a process works. It can also be 
used for making predictions. In terms of intelligent tutoring systems, a model can 
be used effectively to represent student behavior or to demonstrate processes in 
certain domains. However, it is less appropriate for representing knowledge about 
what instructional activities should be performed next 

• Blackboard. Rules pertaining to a particular area of specialization are grouped 
into knowledge sources. Knowledge about the current state is stored in a central 
area known as a blackboard. Whenever a knowledge source has a rule which can 
apply to changing the current state, that knowledge source becomes activated (i.e., 
figuratively speaking, it shouts for attention). More than one knowledge source 
may be activated at any given time. A control module monitors the changes on the 
blackboard, and decides which rules from which knowledge sources should be 
applied and in what order (Nii, 1986). The blackboard approach is particularly 
appropriate for applications requiring multi-level reasoning or flexible control of 
problem solving. However, it has not been widely used, perhaps because: 1) few 
developers have experience with blackboard applications; and 2) there is not much 
commercial software available which has been designed for building blackboard 
applications (Blackboard Technology Group, 1990). 

The current version of the Fundamental Skills Tutor for math uses the list of rules 
approach. However, as more rules are added to reflect increasing knowledge about effective 
teaching, processing the list will gradually become less efficienL Given the number and variety of 
decisions that a teacher must make while tutoring, the blackboard approach is intuitively 
appealing (Bumbaca, 1988; Whitaker & Bonnell, 1992). For example, the blackboard could 
contain knowledge about the current situation (e.g., what has been taught, what approaches have 
been taken) and the student (e.g., student background knowledge, performance on questions so 










far, learning style) Rules could be grouped into knowledge sources based on the types of 
tutoring decisions they contribute to (e.g., when and how to present information, when to change 
topics, what instructional medium to use, when and how to assess student perfc^mance). Finally, 
the control module would choose what rules to apply in deciding what instructional actions to 
take next. 

Implementing a blackboard approach would be difficult, however. First, there are 
currently no tools or shells available for implementing a blackboard architecture on a 
microcomputer (Murray, personal communication. 1991); which means that a blackboard 
implementation would have to be built "from scratch" if it were used on a widely available 
platform. Second, the idea of using a blackboard to dynamically respond to a student is still fairly 
new, so the concept would probably need to be carefully tested before full-scale implementation 
could proceed. Significant progress has already been made in this area by Murray (1989a, 
1989b), using a tool called BBl, on a Symbolics LISP machine; and by Inui, Miyasaka, 
Kawamura & Bourne (1989) using FRANZ LISP, OPS5, PEARL, and Ravors on a Micro VAX 
system. 


Trying to implement a blackboard approach, or exploring other possible system 
organization techniques, would certainly be both challenging and interesting. However, given that 
the fundamental goal of this effort is to identify effective teaching behaviors and get them into a 
readily implementable form, such an exploration could potentially absorb resources which can be 
spent more effectively elsewhere. Since production rules are not incompatible with a blackboard 
approach, a more practical strategy may be to use the "list of rules" approach for the present. 
When it becomes more feasible to use blackboards on microcomputers, the rules can always be 
reorganized. 


Recommended Knowledge Representation Format 

At the present time, a collection of rules seems to be the easiest way to express the 
effective teaching knowledge identified in this study. There are certainly more elegant and 
expressive methods for representing this r^pe of knowledge; in particular, a blackboard type of 
architecture seems very promising. However, it would be difficult to implement this format on a 
microcomputer platform. Given 1) that the ultimate goal of this effort is to produce guidance 
which can be easily implemented in a variety of systems; 2) the popularity of production rules as a 
form of knowledge representation; and 3) the lack of a standard architecture for intelligent 
tutoring systems-the development of a set of tutoring rules is indicated. Such a set of rules could 
be easily modified for use in different intelligent tutoring systems and-after intelligent tutoring 
system development methodologies become more standardized—reworked into a more efficient 
format. 


V. KNOWLEDGE REPRESENTATION DEMONSTRATION 

The goal of this task was to integrate the effective teaching knowledge from Task 1, the 
instructional approach from Task 2, and the knowledge representation format from Task 3 into a 
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set of rules which demonstrate how effective teaching knowledge can be implemented on the 
computer. The purpose was 1) to explore the issues and problems involved in representing 
teaching knowledge; and 2) to produce a starter mle set which can be built upon n future efforts. 


The approach taken included the following steps: 

• Devised method for visually representing effective teaching behaviors 

• Developed list of variables 

• Developed mles 

Devised Method for Visually Representing Effective Teaching Behaviors 

The group of effective teaching behaviors identified during the literature review consisted 
of isolated, disparate pieces of advice. Before these behaviors could be expressed in a computer- 
based form, they needed to be structured into a single, cohesive approach to teaching. Some 
method needed to be devised to show what behaviors were included in the approach, how they 
were sequenced, and the points at which one behavior might be chosen over another. 

In addition to providing visual representation, this method also needed to: 

• reflect the spirit of cognitive apprenticeship 

• be easy to convert to rule form 

• incorporate as much of the research on effective teaching behaviors as possible 

• be as parsimonious as possible (to simplify rule development) 

The research team experimented with a variety of flowcharts and dxision trees. 
Eventually a simple decision tree format was used. The nodes on that decision tree are depicted 
in outline form in Appendix A. Each node (or goal) represents either an action or a decision 
which will impact a later action. 

Issues. Deciding which effective teaching behaviors to represent and how to integrate 
them was the most difficult it of the rule development process. The following paragraphs 
describe some of the issues that nad to be considered and how they were resolved. 

Defining what cognitive apprenticeship includes . The general philosophy behind cognitive 
apprenticeship is sp)elled out in the literature, but the nuts-and-bolts details alx>ut what a cognitive 
apprenticeship approach entails at any given point in a teaching interaction is not. The research 
team decided that this cognitive apprenticeship approach would be an eclectic mix of proven 
instructional methods, with a bias toward showing and doing (rather than telling). This enabled us 
to incorporate as much of the research as possible. 

Lack of appropriate software tools . The ideal software tool would have allowed easy 
representation of teaching actions in graphical form, and then automatic conversion to textual, 
rule-based form. Two software packages which offered these capabilities were reviewed: 
Automated Knowledge Acquisition/Intelligent Authoring Tool (AKAT), and Task Analysis Rule 








Generation Tool (TARGET). However, neither product was complete enough in its current 
version to be helpful. 

In the absence of such software, ordinary drawing software had to be used instead. This 
made the revision process time-consuming and cumbersome—i.e., ever>’ change necessitated 
redrawing the graph-and made no direct contribution to the rule development process. 

Some relevant literature not reviewed . A comprehensive review of all the literature 
related to this effort would include topics in such areas as educational media, instructional design, 
teaching methods, learning theory, and human-computer interface design. However, this type of 
review was not possible within the scope of this project. Because of limited resources, some 
important areas, such as the litera^Mre on screen design and effective use of illustrations, were not 
reviewed at all. 

As stated in Chapter H, however, the team did try to prioritize its investigation of research 
topics based on each topic's degree of usefulness and implementability. Topics for further 
research which are particularly significant or urgent will be noted in the recommendations section 
in the ^nal chapter. 

Deciding what to represent . Even in cases when the team was able to review the relevant 
literature, it was often difficult to apply because the findings were incomplete, speculative, or 
contradictory. For example, some studies on the effectiveness jf advance organizers reponed 
significant improvements in learning with their use, and some did r. jt. Some studies of feedback 
showed a 24-hour delay to be helpful; and some supported the use of immediate feedback. 

In cases where the literature was contradictory, the team identified as many studies as 
possible, and then triangulated the results. In the case of advance organizers, for example, the 
team concluded that they do facilitate learning and retention for most learners. In the case of 
delayed versus immediate feedback, it was found that immediate feedback is helpful in studies 
which simulate actual teaching situations; whereas delayed feedback is more helpful in studies set 
in controlled laboratory environments. 

In cases where the literat'ire was incomplete or speculative, the team made an educated 
guess. For example, cognitive apprenticeship does not sp>ecifically recommend that students 
should be able to review relevant materials after missing a question; nor did the team identify 
research literature which specifically supported or discouraged this practice. However, because it 
is an option that students normally have in traditional instructional settings, it seemed reasonable 
to include it However, we recognize that its inclusion is based on intuitive, rather then empirical 
or theoretical, justifications. 

In some cases, the problem was to decide what not to include. For example, in some of 
the articles on learning strategies (e.g., Danserenu et al., 1975), the authors gave very specific 
prescriptions for what the teacher should say and when, in teaching certain strategies. This raised 
the issue of how specific the rule set should be. 
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The following criteria were developed to determine whether or not to incorporate a given 
teaching action in the rule set; 

1) Is it supported by research? 

Tue team tried to incorporate as many teaching actions supported by research as possible, 
to take full advantage of the knowledge available. 

2) Is it necessary to include this action to present a complete picture of how the cognitive 
apprenticeship approach works? 

For example, diagnosing a student's response was a major goal in the mle set, even though 
the research literature does not specifically address how to do it, because it is something a 
teacher always does. 

3) Is it domain-independent? 

Because the research was intended to provide general guidance, domain specific 
instructional strategies were excluded. 

4) Is it duplicated by some other action or group of actions already included in the 
approach? 

For the sake of parsimony, if an action could be covered by actions already included in the 
teaching approach, then it was not added. In the case of the learning strategies instmction 
mentioned above, for example, the team concluded that the essence of the advice could be 
covered by teaching actions that had already been included. 

5) Is this a design decision? 

Certain types of instructional decisions (such as which instructional medium to rse, 
content, phrasing, and screen layout) were considered to be design decisions, because 
they can only be made at the time a system is being designed. Because the goal of this 
project was to produce generic guidance, rather than a specific system, and because of the 
limited resources available, the research team made little attempt to address these 
decisions. Instead, effort was concentrated on identifying the types of activities and 
content that should be included in the teaching interaction. It was assumed that specific 
decisions about content and media could be made by teachers or instructional designers at 
the time a specific system is being developed. Although this may not be the ideal 
approach, it at least helps to simplify the problem of deciding what to represent 

IDeciding how to represent similar actions . Sometimes teaching actions with similar 
functions have different names. For example, if a student answers a question incorrectly, and the 
teacher respond? with an extended explanation which contains new information, and then follows 
with another question, that could be considered to be providing feedback. However, if this 
response was represented as one type of feedback, then it would duplicate actions dealing with 
providing instruction and eliciting performance. 

In the interests of parsimony, the researchers tried to avoid redundancy as much as 
possible, while still providing a reasonably complete representation of the teaching process. In the 
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example above, feedback was defined to consist of a short explanation at most, after which a 
student would be routed back to Goal 1 (Provide situated instruction). At that point, depending 
on whether or not the student's most recent performance was below a specific criterion level, 
more explanation could be provided if necessary. 

Representing ongoing teaching decisions . The rule set assumes that teaching is a cyclical 
process, with different decisions or activities occurring at different points in the cycle. However, 
there are some types of decisions which human teachers appear to make throughout the cycle, 
rather than at any given point. These decisions are often too general to instantiate in the form of 
rules. For example, human teachers appear to constantly monitor factors such as student fatigue, 
boredom, level of motivation, and the amount of time left to teach, and to adjust what teaching 
activities they select accordingly. It was very difficult to figure out how to represent these types 
of decisions. In addition, there appears to be very little research on how teachers handle this type 
of decision-making. 

Because it was not clear how to represent ongoing teaching decisions, they were left out. 
Perhaps future research will yield more information about the best way of representing these 
situations. 

Developed List of Variables 

In the previous step, the teaching actions that would be included in the set of teaching 
rules were identified. However, in order to write rules, conditions also needed to be defined. A 
human teacher could simply be told "If your student's highly motivated, but hasn't been 
performing well lately, try action X." However, a computer's vocabulary is much more restricted, 
so conditions must be defined in precise terms. 

Appendix B shows the list of instructional variables that was developed to express the 
conditions under which actions would be selected. Each variable represents a type of information 
that is necessary to make a decision. Most variables are set to a default value at the beginning of 
a course, topic, or session. As the student interacts 'vith the computer more, variable values may 
change. 

Issues. The main issue in developing the list of instructional variables was figuring out 
how to deal with variables which were difficult for the computer to measure. A variable was 
considered difficult to measure if 1) there is no good instrument for evaluating it (e.g., 
motivation); or 2) it is difficult to operationalize (e.g., how the student has been performing 
"recently"). 

For variables which were difficult to evaluate, the team made the assumption that the 
computer could obtain the information somehow (e.g., by having a human teachers manually input 
information about each student based on their own "gut" assessments). For variables that were 
difficult to operationalize, the team developed computcr-measurcable variables which 
approximated the desired information. For example, to keep track of a student's recent 
performance, the team developed the LOCPERF variable, which records the student's degree of 
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success on the past few questions. 

Developed Rules 

After instructional variables and teaching actions had been defined, the actual writing of 
the rules was relatively straightforward. Each action on the outline is treated as a goal for the 
computer. A goal becomes the current goal whenever certain conditions arc met. Teaching 
actions occur whenever a goal becomes the current goal. Following is a sample walk-through of 
the rules: 

When a student first logs on. Goal 1 needs to become the current goal. To get there, the 
following rule must fire; 

IF TO?ICPROGRESS = beginniog 

THEN GOAL = provide situated instruction [1] 

TOPICPROGRESS = early 

Translated, this rule says that if a student is just beginning a new topic, then the current goal will 
become to provide situated instruction (Goal 1). In addition, the value of the variable 
TOPICPRCXjRESS is changed to early, indicating that the student has just begun learning about 
the current topic. TOPICPROGRESS is the name of a variable which indicates the degree to 
which the topic being taught has been presented so far. Its value defaults to beginning every time 
a new topic is introduced. 

The next rule that would fire is the following; 

IF GOAL = provide situated instruction [1] 

THEN GOAL = select type of knowledge (1.11 

This rule says that if the current goal is to provide situated instruction (Goal 1), then the current 
goal should become to select what type of knowledge to present (Goal 1.1). The purpose of this 
rule is to move the current goal down from the top level to the level below. 

The next rule that would fire is the following: 

IF GOAL = select type of knowledge [1.1] 

TOPICPROGRESS = early 
THEN GOAL = select domain knowledge [1.1.1] 

This rule says that if the current goal is to select what type of knowledge to teach (Subgoal 1.1), 
and the student has just begun learning about the current topic, then the current goal should 
become to teach knowledge about the domain (Subgoal 1.1.1, as opposed to Subgoal 1.1.2, 
which is knowledge about learning strategies—which the student would have received if he or she 
had been in the topic and struggling for some time already). 

The next rule that would fire is the following; 




IF GOAL = select domain knowledge [1.1.1] 

THEN KTYPE = domain 

This rule causes the value of a variable named KTYPE, which represents the type of knowledge 
which will be taught, to domain. The actual teaching does not get done at this point; that happens 
in a later rule. Notice also that this rule does not change the value of the current goal. This is 
accomplished by the following rule: 

IF GOAL = select domain knowledge [1.1.1] 

THEN GOAL = select method of presentation [1.2] 

This rule says that if the current goal is to select domain knowledge (Subgoal 1.1.1), then the 
current goal should become to select a method of presentation (Subgoal 1.2). Notice that this 
rule gets the system back up one level and advances it to the next subgoal within that level. 

C2omputer-savvy readers may notice that this rule set is not really complete. This rule set 
was developed for demonstration purposes only, to suggest how a rule set might look; it is not 
intended to be executable. 

Issues. The following paragraphs describe issues that had to be addressed during the rule 
development process. 

Context of use unknown . The context in which these rules will ultimately be used was not 
clear. The following assumptions were made for the purposes of the demonstration: 

1) A curriculum, consisting of a list of topics, has already been defined. The rule set 
teaches one topic from this curriculum until it is mastered, then moves to the next topic. 

2) Certain information is readily available, such as background knowledge about the 
student and knowledge about the nature of the subject to be taught In an actual 
implementation of an intelligent tutoring system, this knowledge would need to specified 
for the system before students ever log on. 

3) Information will be collected and updated as needed in the course of a teaching 
session. Although this rule set makes frequent references to what values variables need to 
have before an action can take place, it makes imnimal provision for housekeeping-i.e., 
for initializing those variables or updating them to reflect changes in student performance, 
progress within a topic, and so on. In an actual intelligent tutoring system, programmers 
would need to specify how and when information will be collected and updated. 

4) Decisions about matters such as screen design, reading level, instructional media, 
and how a topic will be presented, are for the most part left to the teacher or instructional 
designer. This set of rules only provides a generic framework which suggests what 
teaching actions should take place, and in what order. More specific teaching decisions 
can only be made when trying to design a system for a particular application. 
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5) A criterion-based approach to instruction is desired. That is, a student will be 
taught until his or her performance exceeds a specified criterion. 

6) Criterion values for variables have been defined. In many cases, this rule set makes 
decisions based on whether the values of certain variables arc above or below criterion. In 
an actual implementation, the range of possible values for these variables would need to be 
defined, and criterion values would need to be specified. 

Indicating completion of goals . In an executable rule set, after firing a rule, the system 
would automatically stan at the top of the rule stack in looking for the next rule to fire. There 
needs to be some way of indicating that a goal has been completed, in order to prevent the system 
from firing the same rule again and again in some cases. The research team tried to provide a 
mechanism for this, but quickly got bogged down. Since this is a housekeeping rather than a 
teaching issue, and because we did not know what software will be used to program these rules 
anyway, we decided to just assume it would somehow be taken care of. 

Addressing all possible conditions . An executable rule set would also somehow 
encompass all possible combinations of goals and conditions; that is, there would always be a rule 
that would fire until it is time for the end of the teaching session. However, the demonstration 
rule set primarily represents infonnation about effective teaching behaviors. Again this is because 
the requirement was only to demonstrate how effective teaching behaviors could be represented in 
rule-based form, not to develop a working set of rules. 

Quantifying criterion values for variables . The educational research literature at best 
indicates what factors are important to monitor, tutd generally what implications different levels of 
those factors have for teaching methods. For example, the literature suggests that as a student's 
mastery of a topic increases, the amount of scaffolding provided by the system should decrease 
(i.e., high mastery requires little or no scaffolding; low mastery requires high scaffolding). 
However, computers need to be told precisely how much scaffolding to provide at precisely what 
level of mastery. Because the application of this general principle may vary over different 
domains or teaching situation ‘he research team did not attempt to quantify levels of values for 
variables. Instead, variable values were referred to as being high, medium, low, above criterion, 
or below criterion. 


VI. CONCLUSIONS AND RECOMMENDATIONS 

The educational research literature is a rich source of effective teaching knowledge, much 
of wJiich is applicable to intelligent tutoring systems. This study was a preliminary attempt to tap 
this knowledge and translate it into a computer-based form. We hope that the outputs from this 
effort will be useful as; 1) information about effective teaching behaviors for both educators and 
ITS developers; 2) lessons learned for others who may wish to use the integrative approach to 
ITS development; and 3) a starting point for developing the tutoring component of intelligent 
tutoring systems. 
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Much more needs to be known about how to individualize instruction. For every 
instructional technique or strategy reviewed for this project, the research about how to target it 
for particular instructional situations was inconclusive. What is most needed is more studies 
which examine how various aptitudes, traits, and treatments interact. These studies could be used 
to tease out the differential effects of variables when they appear in combination. 

It should be noted however, that attempts to individualize instruction should be made with 
caution. Programs which adapt instruction to individual learners have often failed to demonstrate 
a significant improvement over ordinary non-adaptive instruction (Como & Snow, 1986; Doyle & 
Rutherford, 1984). Many of these studies have been conducted with classroom-based, not 
computer-based, programs, and may therefore not be applicable to one-on-one situations. Still, 
ITS developers should remember that more individualization 1) may not necessarily be better, and 
2) may not necessarily be cost-effective. A smart approach may be to initially individualize 
instruction based on variables which have been shown to have a strong correlation with effective 
learning (e.g., I(^. It may also be more cost-effective initially to take Shute's approach of varying 
instructional treatments based on aptitudes, rather than to attempt a model-tracing approach 
(Shute, 1991). Finally, it is important to remember that if the core instructional approach selected 
is either very good or very poor, individualization factors will probably have no significant effect. 

Following are some suggestions for research activities which could build upon the results 
of this study. First, continue reviewing and summarizing effective teaching behaviors from the 
educational research literature, such as screen design, media selection, effective use of visuals 
(e.g., graphics, animation, video), and computer-based assessment methods. Incorporating 
teaching mles from these areas would help an ITS to make more effective decisions about 
presentation of information. 

Second, try instantiating some of the rules and principles identified in this effort within an 
intelligent tutoring system (e.g., the Fundamental Skills Science Tutor). This will help point out 
areas for further refmemenL 

Third, develop an operational rule set. Ideally, this rule set should serve as a domain- 
independent instructional template, which can be dropped into a variety of intelligent tutoring 
systems, with minimal revisions. 

Fourth, consider ways of integrating this rule set with the Advanced Instructional Design 
Advisor (AIDA). Perhaps AIDA could serve as an intelligent front-end to the tutoring template, 
by collecting and structuring domain expertise into a form the template can use. 

Fifth, support research in areas which will enhance the template's capacity to individualize 
instruction. Specifically, 1) identify what the criterion values for doing certain actions should be; 
and 2) describe how constructs such as intelligence, cognitive style, motivation, and self-efficacy 
(and combinations thereof) interact with different types of instructional treatments. 

Sixth, continue to track hardware and software dev lopments which might allow use of 
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more powerful knowledge representation formats, such as blackboard architectures. These would 
make ITSs more capable, and make the task of representing teaching knowledge easier. 

Seventh, develop mle sets for other instructional approaches, such as direct instruction 
and discovery learning. These rule sets can then te instantiated and compared to the cognitive 
apprenticeship approach. 

Eighth, develop techniques to make the tutoring capabilities self-improving (i.e., try to 
utilize machine learning). It would be ideal if the tutor could recognize that certain types of 
teaching actions are ineffective with a particular student, alter its approach accordingly, and then 
use the altered approach subsequently in similar situations. 
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OUTLINE OF TEACHING BEHAVIORS INCLUDED IN RULE SET 

The cognitive apprenticeship model was developed in tree diagram form. Each leaf on the 
tree represented a goal, which is actually a teaching action or decision. The outline below depicts the 
goals and how they relate to one another. 

1. Provide situated instruction 

1.1 Select type of knowledge 

1.1.1 Domain knowledge 

1.1.2 Strategy instruction 

1.1.2.1 Encoding strategies 

1.1.2.2 Monitoring strategies 

1.1.2.3 Affective strategies 

1.2 Select method of presentation 

1.2.1 Model information 

1.2.2 Tell information 

1.3 Select components of presentation 

1.3.1 Instmctional objective 

1.3.2 Advance orgaiuzer 

1.3.3 Concept map 

1.3.4 Mnemonic 

1.3.5 Learner control 

1.4 Select amount of scaffolding 

1.4.1 High 

1.4.2 Medium 

1.4.3 Low 

1.5 Select media components 

1.6 Present instruction 

2. Elicit performance 

2.1 Select problem content 

2.1.1 New problem content 

2.1.2 Repeat/rephrase problem content 

2.2 Select question type 

2.2.1 Recall/knowledge 

2.2.2 Comprehension 

2.2.3 Application 

2.2.4 Analysis 

2.2.5 Synthesis 

2.2.6 Evaluation 
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2.3 Select type of peiformance 

2.3.1 Performance-based question or problem 

2.3.2 Text-based question or problem 

2.3.3 Reflection/Aniculation 

2.4 Present problem 

3. Diagnose student response 

3.1 Collect student information 

4. Provide individualized feedback 

4.1 Do not interrupt student 

4.2 Interrupt the student 

4.2.1 Select content of feedback 

4.2.1.1 Provide information about conectness/incoirecmess of response 

4.2.1.2 Provide correct answer 

4.2.1.3 Provide a short explanation 

4.2.1.4 Provide praiseAeinforcement 

4.2.1.5 Provide performance data 

4.2.1.6 Prompt self-review 

4.2.2 Present response 

4.2.3 Select next goal 

4.2.3.1 Go to Goal 1 

4.2.3.2 Go to Goal 2 

4.2.3.3 Next topic 
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INSTRUCTIONAL VARUBLES USED IN RULES 


Vinable Name 

U«d u> RcDicaeM 

Posadde Values 

Default Values 

ANS 

EvaiuMion of madaiti annva to the mo* leoenily 
asked qoeauoa 

Correct, urcoirect 

None (vanes) 

ANX 

Studem'i level of anxiety abota leaminc 

Above or below cnlenon 


AO 

Whether or not an advance or|anizer will be used ui 
presenciof aismicaon at a panicular point in lone. 
Based on lenihs from Subcoal I.3J. 

Yes.no 

Yes 

APT 

Student aptitude 

Above or below cnierion 

Can be deutmined for each 
snideic prior to ctsuuctian 
based on teacher mpia or 
tenant 

CA 

Whether or not feedback will indude the conect 
ansaaer to the questioa Bated on lesuiit hom 
Subsoal 4.Z1.2. 

Yes, no 

No 

CUNPO 

Whether or not to provide infoimatioo about the 
oonecmess or inoomclneas of a leqponae Bated on 
Ksulu from Subcoal 4.2.1.1. 

Yes, no 

No 

CM 

Whether or not a concept map will be used in 
preaeotinc insnuciioo at a particular point in tone. 
Based on results from Subcoal 1.3.3. 

Yes, no 

No 

CURRTOPIC 

The tiudem't place in the cuniculum. in lenns of 
what topic he/the it on. 

Prtmout topic: topic i-l 

CurTtni topic: topic x 

Next topic: topic x-il 

Not applicable 

GOAL 

The cunent goal 


Not applicable 

HISTPERF 

Ovtrall toident peiformance in the cumculum at 
any given point in time. This variable records how 
well a student has performed over a longer period of 
lane (e^.. over aevenl topics within the same 
cuniculum). 

Above or below cntenon 

Below cnieiicn 

KTYPE 

Type of knowledge tekcrod to be presented. Based 
on results from Subcoal 1.1 and associated leaves. 

Domain, enoodoig, monitonitg. 
aCTeciive 

Domain 

LC 

Whether or not learner ooniroi will be allowed 
during insliuctian. Based on results from Subgoal 
1.3.5. 

Yes.no 

No 

LOG 

Student's locus of control 

Iniemal, exiemal 

Can be determined for each 
snideiu prior ir hstruction 
bated on teacher input or 
testinc. 

LOCPERF 

Student peiformance on the most recent series of 
quettioiu 

Above or below ciilerion 

Below criterion 

METHOD 


Model, tell 

McxU 

MN 

Whether or not a nmemonic will be used in 
presenting innniction at a panicular point in time. 
Based on results from Subcoal 1.3.4. 

Yes, no 

No 

MOTIVLEARN 

Gemal motivation to leam (trait motivation) 

Above or below criterion 

Can be determined for each 
tnident prior to inttruction 
based on teacher input or 
testaut. 

MOnVTOPIC 

Motivation to learn a particular topic (state 
motivation) 

Above or below criterion 

Can be determined for each 
student prior to inttrtiction 
bored on teacher inpul or 
lettinc. 

NUMOUEST 

Number of questions asked so far 

Numerical value 

None 

OBJ 

Whether or not an instruciional objective will be 
used in presenting insliuciion at a particular point ir. 
time, ^sed on result from Subeod 1.3.1. 

Yes, no 

Yes 

PD 

Whether or not to provide performance dau. Based 
on tesuh from Subgoal 4.2.I.3. 

Yes, no 

No 
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INSTRUCTIONAL VARIABLES USED IN RULES (CONT.) 


Vinible Name 


Possible Values 

Default Values 

PE 

Whether or not to provKle an eapUnaiion. Based on 
result from Subsoal 4.2.1.3. 

Yea, no 

No 

PERFDIFF 

Lxvel of difficulty of peifomianoe 

Easy, iiueimediate, difficult 

Vanes. Can be detemuned 
for each problem either by 
■he designer or by some 
computer algontlim. 

PERFSTATE 

Whethet or not a oudeiu ii aiill vraiking on a 
problem at a the tune (Le., whether he/ihe it done 
with it, or it nill workmi; on it). 

SiUI working, done 

Done 

PR 

Whether or not to provide icinforoemenL Bated on 
retult from Subgoal 4.2.1.4. 

Yes, no 

No 

PTYPE 

The type of performance that will be requited ffom 
the itudenL Based on results from Subgoal 23 and 
assoaated leaves. 

Performance-based, text-based, 
rcflecttan/itticuJaiion 

Performanoe-based 

PTYPEPREV 

The PTYPE of the last quesuon (i.e., most recent) 
asked. 

Petfotmance-based. text-based, 
reflection/aniculaiton 

Vanes 

QCONTENT 


New, same 

New 

QTYPE 

Type of quesUon the student will be asked. Based on 
results from Subgoal Z2 and associated leaves. 


RecalL/know ledge 

QTYPEPREV 

The QTYPE of the last quesuon (i.e., tiKist recent) 
asked. 

RecallArwwIedge, 
comptehetuion, applicauon, 
arulvsis, synthetii, evaluauon 

Vanes 

REQAO 

Wliether or not the learner requesu an advance 
organizer. Based on results from Subgoal 1.3.2. 

Yes.tro 

No 

REQCM 

Wliether or not the learner requests a concept map. 
Based on results from Subgoal 1.3.3. 

Yes, i» 

No 

REQMN 

Whether or not the learner requests a mnemonic. 
Based on results from Subgoal 1.3.4 

Yes, no 

No 

REQMODEL 

Whetho or not the learner requesu a demonstration 
of prxxsess or skill. 

Yes, no 

No 

reqobj 

Whether or ika the learner requests an mstructional 
objective 

Yes, no 

No 

REQREPEAT 

Whether or not student has requested that a question 
be repealed 

Yes, no 

No 

reoreview 

Whether or not student has requested a review 

Yes.no 

No 

RESPCERT 

Student response certitude for (he iTK>st recent 
quesuon 

Above or below cnienon 

Vanes. Determined by 
asking students how certain 
thev are of their responses. 

RESPCERT(/opiC x) 

SuKleni's avenge response oertiuide on topic so far 

Above or below critenon 

Vanes. Determined by 
nudents’ average response 
ceitiiude after aruwering 
several rrueslions. 

SCAFF 

Amount of scaffoldmg that will need to be provided 
during instiuctioa Based on results from Subgoal 

1.4 and associated leaves. 

High, medium, low 

High 

SES 

Student socioeomomic status 

Above or below criterion 

Can be detetmirKd for each 
student prior to msmiction 
based on teacher input or 
ic sting 

SR 

Whether or not to prompt a self.review. Based on 
result from Subgoal 4 2.1.6. 

Yes, no 

No 

SRECALL 

Student performance on questions requiring recall of 
information 

Above or below cnienon 

Critenon. 

TESTPERF(iopic x) 

Student performaiKC on fmal test for mastery of 
topic X 

Above or below entenon 

Vanes, depending on 

student perfoimsnoc. 
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INSTRUCTIONAL VARIABLES USED IN RULES (CONT.) 


1 Viriable Nune 

Used to Represent 

Possible Values 

Default Values 

1 TIMELEFT 

Amount of dstk left in clast or session 

Above or below criterion 

None (vanes) 

1 TIMEWAITING 

Ai.ioiau of time spent watting for inpsrt from sntdent 
(while elicidna performanoe) 

Above or below cnienon 

None (varies) 

TOPICCHAR(3) 

(1) = nujuiiet manorizaiion 

of new tenm or 
conccfls 

(2) = can be denwfutnted or 

imdelled (e.g., a 
process, ilcill, or 
procedure) 

(3) = includes inter-relaied 

concepts 

CSiatactehttics of topic. This is an array variable. 
Eadi elemem of the array tepreaents the presence or 
abtetKX of oieaspeaof the topic. Aiopiccanbe 
characterized by a combination of yet and no 
attswen for each elemetu. 

Each element may assume one of 
iwc' vilues-eilher yes or no. 

t-*-. 

TOPtCCHAR(l) = yei means 
that the topic tequiiei 
memorization 

TOPICCHAR(3) = Romeatu 
that the topic does not mclude 
inter-relaied concepu 

This infotnuujon would 
need lo be specified for 
each topic within the 
curriculum during the 
instiuclional design 
process, and loaded m each 
lane the student changes to 
a different topic. 

TOPICPERF(wpics) 

Student's level of mastery of a topic at any given 
time. This variable expresses the idea ofhow 
ouidcly the student is catching on 

LmJe or no pnrgreasion, nomul 
progression, mastery 

LinJe or no progression 

TOPICPERFCHAR(6) 

(1) = recall 

(2) = oomptetsension 

(3) = applicauon 

(4) = arralysii 

(5) = synthesis 

(6) = evaluation 

Characteristics of the types of performance tequued 
by a topic Each element of the array represenu the 
presence or absence of a type of performanoe. A 
topic can be characterized by a combination of yes 
artd no answers for each element 

Each elemetu may assume one of 
two vtlues-either yes or no. 

E.g.. 

TOPICPEliFCHAR(2) = yti 
means that comprtkenston 
needs lo be assessed when 
dialing performance with the 
siudenL 

This information would 
need to be specified for 
each topic within the 
curriculum during the 
instiuctional design 
process, and loaded m each 1 
lime the sUident changes to | 
a diiTerent topic. | 

TOPICPROGRESS 

[>egree lo which topic hAS been covered thus far 

This variable eitpresses the idea of how much of the 
topic maiehal has been covered so far and how 
much is left 

Begintung, early, midway, 
nearly complete, complete 

Beginning 

TRIES 

Number of tries s student his been given to inswer 
the current question. 

Above ditehon, below entenon 

Zero 
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Notes on the Sample Rule Set 


The purpose of the sample rule set is to illustrate how rules might be written to execute 
effective teaching behaviors within a cognitive apprenticeship instructional approach. The 
emphasis is on demonstrating how the effective teaching behavior knowledge identified in this 
study could be expressed in rule form. The emphasis is on developing a comprehensive or 
executable rule set. For example, to be implemented within an instructional module, "house 
cleaning rules" would need to be added to keep the program from falling into infinite loops. 

In developing these rules, the following assumptions were made about the context in 
which they would be used: 

• A curriculum, consisting of a list of topics, has already been defined. The rule 
set teaches one topic from this curriculum until it is mastered, then moves to the 
next topic. 

• Certain information is readily available, such as background knowledge about 
the student and knowledge about the nature of the subject to be taught. In an 
actual implementation of an intelligent tutoring system, this knowledge would 
need to specified for the system before students ever log on. 

• Information will be collected and updated as needed in the course of a teaching 
session. Although this rule set makes frequent references to what values 
variables need to have before an action can take place, it makes minimal 
provision for housekeeping--i.e., for initializing those variables or updating 
them to reflect changes in student performance, progress within a topic, and so 
on. In an actual intelligent tutoring system, programmers would need to specify 
how and when information will be collected and updated. 

• Decisions about matters such as screen design, reading level, instructional 
media, and how a topic will be presented, are for the most pan left to the 
teacher or instructional designer. This set of rules only provides a generic 
framework which suggests what teaching actions should take place, and in what 
order. More specific teaching decisions can only be made when trying to design 
a system for a particular application. 

• A mastery approach to instruction is desired. That is, a student will be taught 
until his or her performance exceeds a specified criterion. 

• Criterion values for variables have been defined. In many cases, this rule set 
makes decisions based on whether the values of certain variables are above or 
below criterion. In an actual implementation, the range of possible values for 
these variables would need to be defined, and criterion values would need to be 
specified. 
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Rules for Providing Situated Instruction (Goal 1) 

Goal 1: Provide situated instnicdon 

Function: This goal (and its sub-goals) is fired to present instruction. It addresses the information presentation 
aspects of instruction-when the student is told or shown something. The amount presented depends on the "grain 
size" of the instruction. It can be very minimal - e.g., giving instructions on how to use an instructional game or 
simulation. Or, it can be more detailed and complex - e.g., having the student watch a video or read a long 
explanation. 

Reasons for Goal Selection: 

1) If the student is beginning a new topic. 

Example rule: 

IF TOPICPROGRESS = beginning 
THEN GOAL = provide situated instruction [IJ 
TOPICPROGRESS = early 

Explanation: This rule makes Goal 1 the current goal, and sets the value of the variable 
TOPICPROGRESS to "early" to reflect that instruction on that topic has already begun. 

2) If the student needs a fresh presentation of the current topic (or if student needs to leam a new sub-topic 
within the same topic) and there is time left 

Example rule: 

IF GOAL = reset variables to return to goal 1 [423.1] 

TOPICPROGRESS - early OR midway OR nearly complete 
TIMELEFT > criterion 

THEN GOAL = provide situated instruction [1] 

Explanation: This rule assumes that the student has already completed one cycle of instruction (i.e., from 
Goal 1 through Goal 4) and has reached Subgoal 4.2.3.1, which means that the student should pass 
through another cycle on the same topic. If TOPICPROGRESS is still continuing, and there is time left, 
then Goal 1 becomes the current goal. 


Outcome: A decision to provide situated instruction. 

Next Step: Establish parameters for type of knowledge to be taught (Subgoal 1.1), method of presentation (Subgoal 
1.2), components of presentation (1.3), amount of scaffolding (1.4), and delivery medium (1.5). Present instruction 
based on parameters (1.6). 
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Subgoal 1.1: Select type of knowledge 

Function: This goal is fired to detennine whether domain knowledge or learning strategies knowledge should be 
{sesented. 

Reason for Goal Selection: 

1) If Goal 1 is the current goal. 

Example rule: 

IF GOAL s provide situated instruction (IJ 
THEN GOAL = select type of knowledge flJJ 

Explanation: If the current goal is Goal 1, then the current goal should become Subgoal 1.1. 

Outcome: A decision to select what type of knowledge to teach. 

Next Step: Either 1.1.1 (domain knowledge) or 1.1.2 Oearning strategies). 
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Subgoal 1.1.1: Select domain knowledge 


Function: This goal is fired to select domain knowledge instruction. 

Reasons for Goal Selection: 

1) When a new topic is first being presented. 

Example rule: 

IF GOAL = select type of knowledge 11.1 ] 

TOPICPROGRESS = early 
THEN GOAL = select domain knowledge [1.1.1) 

2) If a student is making satisfactory progress on a topic (as evidenced by student historical performance). 
Example rule: 

IF GOAL = select type of knowledge [1.1} 

TOPICPROGRESS = rmdway OR nearly complete 
HISTPERF > criterion 

THEN GOAL = select domain knowledge [1.1.1J 
Outcome: A decision to teach domain knowledge. 

Execution: Another rule is needed to set the knowledge type parameter (KTYPE) to "domain knowledge.” For 
example, 

IF GOAL - select domain knowledge [ 1.1.1) 

THEN KTYPE = domain 

Next Step: Select method of presentation (Subgoal 1.2). 
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Subgoal 1.1 Jt; Select strategy instruction 


Function: This goal to select instruction on learning strategies, such as encoding strategies, comprehension 
monitoring strategies, or affective strategies. 

Reason for Goal Selection: 

1) The student has not been performing very well on questions related to the topic. 

Example Rule: 

IF GOAL = select type of knowledge 11.1] 

TOPICPROGRESS - midway OR nearly complete 
HISTPERF < criterion 

THEN GOAL - select strategy instruction 11.12] 

Outcome: A decision to teach learning strategies. 

Next step: Decide what type of learning strategies to teach (Subgoal 1.12.1,1.1.22, or 1.1.2.3). 
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Subgoal Teach encoding strategies 

Function: This goal is fired to teach the student strategies for remembering information better. 

Reasons for Goal Selection: 

1) If (a) the goal is to provide strategy instruction, (b) the topic is one that requires memorization, and (c) the 
student has not been performing well on items requiring recall of information. 

Example rule: 

IF GOAL = provide strategy instruction (1.12] 

TOPICCHAR( 1) = yes (i.e., topic requires memorization) 

SRECALL < criterion 

THEN GOAL = teach encoding strategies [1.12.1] 

2) If (a) the goal is to provide strategy instruction, and (b) there is no specific reason to choose one strategy 
over another. 

Example rule: 

IF GOAL = provide strategy instruction 11.12] 

GOAL {1.12.1.1.122, or 1.12.3} not selected 
THEN GOAL = teach encoding strategies {1.12.1] 

Outcome: A decision to teach encoding strategies. 

Execution: Another rule is needed to set the Knowledge Type parameter to "encoding strategies." For example, 

IF GOAL = teach encoding strategies 11.12.1] 

THEN KJYPE = encoding 

Next step: Select method of presentation (Subgoal 1.2). 
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Subgoal 1.1J2J1: Teach moDitoring strategies 

Function; This goal is fired to teach the student to recognize when he or she is not understanding. 

Reason for Goal Selection: 

1) If the student is not performing well on a topic, but thinks that he/she is (as indicated by his/her response 

certitude). 

Example rule: 

IF GOAL = provide strategy instruction (1.12} 

TOPlCPERF(topic x) - little or no progression 
RESPCERT > criterion 

THEN GOAL = teach monitoring strategies [1.122] 

Outcome; A decision to teach monitoring strategies. 

Execution: Another rule is needed to set the Knowledge Type parameter to "monitoring strategies." For example, 

IF GOAL = teach monitoring strategies (1.12 2] 

THEN KTYPE = monitoring 

Fiorn here, you should go to Subgoal 12. (The rule to do so is specified under Subgoal 1.2). 


113 






Subgoai l.UJ: Teach affective strategies 

Function: This goal is fired to teach students strategies for dealing with debilitating emotions, such as test anxiety. 
Reasons for Goal Selection: 

1) If it is known that the student is highly anxious. 

Example rule: 

IF GOAL = provide strategy instruction [1.12] 

ANX > criterion 

THEN GOAL = teach affective strategies [1.123] 

2) If the student seems to know the topic (based on informal performance measures) but does poorly on tests. 
Example rule: 

IF GOAL = provide strategy instruction [1.12] 

TOPICPFRF(topic x) > criterion 
TESTP ERF (topic x) < criterion 
THEN GOAL = teach affective strategies [1J23] 

3) If the student lacks motivation to learn a particular topic (state motivation). 

Example rule: 

IF GOAL - provide strategy instruction [1.12] 

MOTfVTOPIC < criterion 

THEN GOAL - teach affective strategies [1.123] 

4) If the student lacks motivation to learn in general (trait motivation). 

Example rule: 

IF GOAL = provide strategy instruction [1.12] 

MOTIVLEARN < criterion 
THEN GOAL = teach affective strategies [1.123] 

Outcome: A decision to teach affective strategies. 

Execution: A rule is needed to set the Knowledge Type variable to "affective strategies." For example, 

IF GOAL = teach affective strategies [I.I23] 

THEN KTYPE = affective 

Next Step: Select method of presentation (Subgoal 12). 
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Subgoal IJl: Select method of presentatioii 

Function: This goal is flred to decide whether to model or to tell information. 

Reasons for Goal Selection; 

1) If the Knowledge Type parameter has been set 
Example rule: 

IF KTYPE 0 

THEN GOAL b select method of presentation {12} 

Outcome: Decision to select method of presentation. 

Next step: Decide whether to model knowledge (Subgoal 1.2.1) or to tell knowledge (Subgoal 1.2.2). 
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Subgoai 0.1: Model knowledge 

Function: This goal is fired to model knowledge (i.e., teach by showing or demonstrating). 

Reasons for Goal Selection: 

1) If instruction is just beginning and if the topic lends itself to being demonstrated or modeled. 

Example rule; 

IF GOAL = select method of presentation 112] 

TOPICPROGRESS = early 

T0PICCHAR(2) = yes (i.e., topic can be demonstrated or modeled) 

THEN GOAL = model knowledge 112.1] 

2) If the student requests a model. 

Example rule: 

IF GOAL = select method of presentation [12] 

TOPICPROGRESS = continuing 

T0P1CCHAR(2) = yes (i.e., topic can be demonstrated or modeled) 

HISTPERF > criterion 
REQMODEL = yes 

THEN GOAL = model knowledge [12.1] 

Outcome: A decision to present instruction using a modeling approach. 

Execution: Another rule is needed to set the Teaching Method parameter (METHOD) to "model." For example, 

IF GOAL - model knowledge [12.1] 

THEN METHOD = model 

Next Step; Select components of instructional presentation (Subgoal 1.3). 





Subgoal 1.Z2: Tell knowledge 


Function: This goal is Gred to tell knowledge. 

Reasons for Goal Selectioo: 

1) If the topic requires menKuization of terms or concepts. 

Example rale: 

IF GOAL « select method of presentation 112] 

TOPICPROGRESS » continuing 

TOPICCHAR(l) * yes (Le., topic requires memoruation of new terms or concepts) 
THEN GOAL = tell knowledge [122] 

2) If the student has historically performed well on this topic and requests a review of the material. 
Example rule: 

IF GOAL = select method of presentation [12] 

TOPICPROGRESS^ beginning 

TOPlCCHAR( 1) = yes (i.e., topic requires memorization of new terms or concepts) 
HISTPERF > criterion 
REQREVIEW = yes 

THEN GOAL = tell knowledge [122] 

Outcome: A decision to present information to the sradent by telling the student 

Execution: Another rale is needed to set the Method of Presentation (METHOD) parameter to "tell." For 
example, 

IF GOAL = teU knowledge 
THEN METHOD^ tell 

Next Step: Select components of instructional presentation (Subgoal 13). 
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Subgoal U: Select components of instructional presentation 

Function: This goal is fired to select which components should be included in an instructional presentation-e.g., 
instructional objective, advance organizer, conc^t map, etc. 

Reasons for Goal Selection: 

1) If Method of Presentation has been selected 
Example Rule: 

IF METHOD * 0 

THEN GOAL = select components of instructional presentation [13] 

Outcome: A decision to select components of instructional presentation. 

Next step: Select components to include, such a? instructional objectives (Subgoal 1.3.1), advance organizers 
(Subgoal 1.3.2), concept maps (Subgoal 1.3.3), mnemonics (Subgoal 1.3.4), and learner control (Subgoal 1.3 J). 
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Subgoal U.l: Provide an instructional objective 
Function: This goal is fired to provide an instructional objective. 

Reasons for Goal Selection: 

1) If just beginning presentation of a topic 
Example Rule: 

IF GOAL s select components of instructional presentation 
TOPICPROGRESS = early 

THEN GOAL = provide instructional objective [13.1 / 

2) If beginning a new sub-topic within a topic 
Example Rule: 

IF GOAL - select components of instructional presentation [13] 

TOPICPROGRESS = midway OR nearly complete 
THEN GOAL = provide instructional objective (proximal) [13.1J 

3) If the student asks to see the objective 
Example Rule: 

IF REQOBJ = yes 

THEN GOAL « provide instructional objective 
Outcome: A decision to present an objective. 

Execution: Another rule is needed to set the Present Objective (OBJ) parameter to "yes.” For example, 

IF GOAL = provide instructional objective 
THEN OBJ ^ yes 

Next Step: See if any other components should be selected (Subgoals 1.32, 1.3.3,1.3.4,1.3.5) or select amount of 
scaffoldbg (Subgoal 1.4). 
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Subgoal 1Provide an advance organizer 


Function; This goal is Hred to provide an advance organizer. 

Reasons for Goal Selection; 

1) When a new topic is first presented; 

Example Rule; 

IF GOAL = select components of instructional presentation [13] 

TOPiCPROGRESS = early 

THEN GOAL = provide an advance organizer [132] 

2) If the student requests an advance organizer 
Example Rule; 

IF GOAL = select components of instructional presentation [13] 

REQAO = yes 

THEN GOAL = provide an advance organizer [132] 

Outcome; A decision to present an advanced organizer. 

Execution; Another rule is needed to set the Advanced Organizer (AO) parameter to "yes." For example. 

IF GOAL = provide an advance organizer [132] 

THEN AO = yes 

Next Step: See if any other components should be selected (Subgoals 1.3.2,1.3.3.1.3.4,1.3.5) or select amount of 
scaffolding (Subgoal 1.4). 
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Subgoal UJ: Provide a concept map 
Function: This goal is Hred to provide a concq)t map. 

Reasons for Goal Selection: 

1) If student is just beginning a new topic and that topic lends itself to concept mapping. 

Example Rule: 

IF GOAL = select components of instructional presentation [1.3] 

TOPICPROGRESS - beginning 

TOPICCHAR(3) = yes (Le.. topic includes inter-related concepts) 

THEN GOAL = provide concept rruip [133] 

2) If the student requests a concept map. and the topic lends itself to concept mapping. 

Example Rule: 

IF GOAL - select components of instructional presentation [13] 

T0PICCHAR(3) = yes (i.e., topic includes inter-related concepts) 

REQCM = yes 

THEN GOAL = provide a concept map [133] 

Outcome: A decision to provide a concept map. 

Execution: An additional rule is needed to set the Concept (CM) parameter to "yes." For example, 

IF GOAL = provide a concept map [133] 

THEN CM = yes 

Next Step: See if any other components should be selected (Subgoals 1.3.2,1.3.3,1.3.4,1.3.5) or select amount of 
scaffolding (Subgoal 1.4). 
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Subgoal U.4: Provide a moemoDk 


Function: This goal is fued to provide a mnemonic. 

Reasons for Goal Selection; 

1) If the topic requires memorization of new terms or concepts. 

Example Rule: 

IF GOAL = select components of instructional presentation [13] 

TOPICCHAR(l) = yes (i.e., topic requires memorization of new terms or concepts) 

THEN GOAL = provide a mnemonic [13.4] 

2) If the student requests a mnemonic. 

Example Rule: 

IF GOAL = select components of instructional pre.^ntation [13] 

REQMN = yes 

THEN GOAL = provide a mnemonic [13.4] 

Outcome: A decision to provide a mnemonic. 

Execution; Another rule is needed to set the Mnemonic parameter (MN) to "yes." For example, 

IF GOAL = provide a mnemonic [13.4] 

THEN MN^yes 

Next Step: See if any other components should be selected (Subgoals 1.3.2,1.3.3,1.3.4,1.3.5) or select amount of 
scaffolding (Subgoal 1.4). 
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Subgoal U^: Allow learner control 


Function: This goal is fired to allow learner control. 

Reasons for Goal Selection: 

1) If the student is a high aptitude learner. 

Example rule: 

IF GOAL - select components of instructional presentation [13J 
TOPICPROGRESS = early OR nudway OR nearly complete 
APT > criterion 

THEN GOAL « allow learner control of some features of instruction [ 1JJ] 

Outcome: A decision to allow learner control 

Execution: Another rule is needed, to set the Learner Control (LQ parameter to ’yes." For example. 

IF GOAL = allow learner control of some features of instruction 11.35] 

THEN LC = yes 

Next Step: See if any other components should be selected (Subgoals 1.3.2,1.3.3,1.3.4,1.3.5) or select amount of 
scaffolding (Subgoal 1.4). 
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Subgoal 1.4: Select amount of scafTolding 

Function: This goal is fired to select the amount of scaffolding (high, medium, or low) to provide. 
Reasons for Goal Selection: 

1) If all available components of an instructional presentation have already been considered (but not 
necessarily selected). 

Example rule: 

IF GOAL s select the components of instructional presentation [13] 

THEN GOAL s select the amount of scc^lding [1.4] 

Outcome; A decision to select the amount of scaffolding to provide. 

Next step: Decide whether to provide high (1.4.1), medium (1.4.2), or low (1.4.3) scaffolding. 
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Subgoal 14.1: Select high scafTolduig value 


Function: This goal is fired to {Hovide high scaffolding. 

Reasons for Goal Selection: 

1) If the student has made little progress through the ttqiic. 

Example rule: 

IF GOAL s select amount efset^oUUng 11.4] 

TOPICPERF(topic x) = little or no progress 
THEN GOAL s select high scaffoMng value 11.4] 

Outcome: A decision to provide high scaffolding. 

Execution: A rule is needed to set the Scaffolding (SCAFF) parameter to‘high.” For example, 

IF GOAL s select high scaffolding value [1.4.1] 

THEN SCAFF^high 

Next stqi: Select media components (Subgoal 1.5). 
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Subgoal 1.4 J: Select medium scafTolding value 
Function: This goal is fired to provide medium scafTolding. 

Reasons for Goal Selection: 

1) If the student is midway through the topic and is progressing at a normal rate of speed. 

Example rule: 

IF GOAL = select amount of scaffolding 11.4] 

TOPICPERF(topic x) = normal progression 
THEN GOAL = select medium scaffolding value [1.4] 

Outcome: A decision to provide a medium level of scaHblding. 

Execution: Another rule is needed to set the Scaffolding (SCAFF) parameter to "medium." For example, 

IF GOAL - select medium scaffolding value (1.42] 

THEN SCAFF = medium 

Next step: Select media components (Subgoal 1.S). 
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Subgoal 1.4 J: Select low scaffolding value 

Function: This goal is fued to provide low scaffolding. 

Reason fw Goal Selection: 

1) Student has nearly mastered a topic. 

Example rule: 

IF GOAL = select amount of scaffolding [1.4} 

TOPICPERF( topic x) = mastery 
THEN GOAL = select low scaffolding value [1.4. 3] 

Outcome: Decision to provide a low level of scaffolding. 

Execution: Another rule is needed to set the Scaffolding (SCAFF) parameter to ’low.’ For example, 

IF GOAL = select low scaffolding value [1.43] 

THEN SCAFF = low 

Next step: Select media components (Subgoal 1.5). 
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Subgoal 1^: Sdcct media components 

Function: This goal is fired to select media components. 

Reasons for Goal Selection: 

1) If amount of scaffolding has been selected. 

Example rule: 

IF SCAFF * 0 

THEN GOAL = select media components [15} 

Outcome: A decision to select media components. 

Execution: A set of rules for selecting appropriate, instrucdonal media (e.g., text, video, animation, graphics, 
audio, etc.) needs to be inserted here. An example rule fat accessing this set follows. 

IF GOAL = select media components 

THEN GOAL = select appropriate instructional media 

Next step: Present mstruction (Subgoal 1.6) 
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Subgoal 1.6: Present situated instruction 

Function: 1 his goal is fired to present situated instruction according to the parameters established by the previous 
sub-goals. 

Reasons for Goal Selection: 

1) If instructional media have already been selected. 

Example rule; 

IF instructional media have been selected 

THEN provide situated instruction within parameters specified by KTYPE, METHOD, OBJ, 

AO, CM. MN, LC, SCAFF, and metUa components 

Outcome; Instruction presented according to established parameters. 

Execution: Some mechanism for using these parameters to decide how to present instruction needs to be 
developed. 

Next step: Test soident knowledge by eliciting student performance (Goal 2). 
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Rules for Eliciting Performance (Goal 2) 
Goal 2: Elicit performance 


Function: This goal (and its associated sub-goals) is fired when there is a need to elicit performance firom the 
student The purpose of eliciting performance may be either to test or to give the student an o^xirtunity to learn by 
doing. 

Reasons for Goal Selection: 

1) If situated instruction has been provided. 

Example rule: 

IF situated instruction ’uxs been provided 
THEN GOAL = elicit performance [2] 

2) If the student needs more practice to master a topic. 

Example rule: 

IF GOAL = reset variables to return to Goal 1 [4.232] 

TIMELEFT > criterion 

TOPICPROGRESS = early OR midway OR nearly complete 
THEN GOAL = elicit performance [2] 

Outcome: A decision to elicit student performance. 

Next step: Select problem content (Subgoal 2.1), question type (Subgoal 2.2), and performance type (Subgoal 2.3). 
Present problem (Subgoal 2.4). 
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Subgoal 2.1: Select problem contcot 


Function; This goal is fired to select whether the omtent of a qiKstion or problem should be new or old. 

Reasons for Goal Selection: 

1) If current goal is to elicit performance. 

Example rule: 

IF GOAL = elicit performance 
THEN GOAL = select problem content [2.1] 

Outcome: A decision to select {noblem content 

Next step: Decide whether to present a new problem (Subgoal 2.1.1.) or to le-present/rtphrase a previous problem 
(Subgoal 2.1.2). 


131 



Subgoal 2.1.1: New problem cootent 
Function: This goal is fired to select new problem content 
Reasons for Goal Selection: 

1) If the student answered the last problem correctly and has performed satisfactorily on the last few 
problems on a topic. 

Example rule: 

IF GCAL = select problem content [2.1] 

ANS * correct 
LOCPERF > criterion 
TOPICPERF(topic x) = normal progression 
THEN GOAL = new problem content {2.1J] 

Outcome: A decision to present new problem content 

Execution: Another rule is needed to set the problem content (QCONTENT) parameter to’new.’ For example. 

IF GOAL - new problem content [2.1.1] 

THEN QCONTENT = new 

Next step: Select question type (Subgoal 2.2). 
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Subgoal 2.U: Repeat/rcphrase problem content 


Function: This goal is fired to repeat or rephrase problem content. 

Reasons for Goal Selection: 

1) If the student answered the last problem inctmectiy. 

Example rule 

IF GOAL select problem content [2.1] 

ASS - incorrect 
LOCPERF > criterion 
TOPICPERF(topic x) «= normal progression 
THEN GOAL = repeat/rephrase problem content {2.12] 

2) If the student requests a review of the previous problem. 

Example rule: 

If GOAL - select problem content pt.IJ 
REQREPEAT = yes 

THEN GOAL - repeat/rephrase problem content f2J 2] 

Outcome: A decision to repeat or rephrase a i»oblem. 

Execution: Another rule is needed to set the Content (QCONTENT) parameter to ’same.” For example, 

IF GOAL = repeat/rephrase problem content {2J2J 
THEN QCONTENT ^ same 

Next step: Select question type (Subgoal 2.2). 
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Subgoal 2 J; Select question type 


Function: This goal is fired to select what type of question to ask (e.g., recall/knowledge, comprehension, 
application, analysis, synthesis, or evaluation). 

Reasons for Goal Selection: 

1) If problem content has been selected. 

Example rule: 

IF QCONTENT * 0 

THEN GOAL = select question type [22J 

Outcome: A decision to select the type of question to 

Next step: Select one of the following question types - Rccall/knowledge (Subgoal 2.2.1), Comprehension (2.2.2), 
AppUcadon (2.2.3), Analysis (2.2.4), Synthesis (2.2.5), or Evaluation (2.2.6). 
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Subgoal 2^.1: Recall/knowledge 


Function: This goal is fued to ask a recall/knowledge question. 

Reasons fw Goal Selection: 

1) If the previous question type was recaU/knowledge and the question content is the same. 

Example rule: 

IF GOAL K select question type [22J 
QTYPEPREV - recalUknowledge 
QCONTENT = same 

THEN GOAL ^ recalUknowledge [22 J] 

2) If the student is just beginning the topic and performing the topic requires recall. 

Example rule: 

IF GOAL = select question type [22] 

TOPICPROGRESS = beginning 

T0PICPERFCHAR(1) = yes (i.e.. topic performance includes recall) 

THEN GOAL = recalUknowledge [22.1] 

Outcome: A decision to ask a recall^owledge question. 

Execution: Another rule is needed to set the Question Type (QTYPE) parameter to "recaU/knowiedge." For 
example, 

IF GOAL = recalUknowledge [22.1] 

THEN QHPE^ recalUknowledge 

Next step: Select type of performance (Subgoal 2.3). 
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Subgoal 2.12: ComprehensioQ 


Function: This goal is fired lo ask a comprehension question. 

Reasons for Goal Selection: 

1) If the previous question was a comprehension question and the content is the same. 

Example rule: 

IF GOAL ~ select question type [22J 
QTYPEPREV = comprehension 
QCONTENT = same 
THEN GOAL = comprehension [222] 

2) If the previous question was a recall/knowledge qiKStion and the content is new. 

Example rule: 

IF GOAL = select question type [22] 

QTYPEPREV = recalllknowledge 
QCONTENT = new 

THEN GOAL - comprehension [222] 

Outcome: A decision to ask a comprehension question. 

Execution: Another rule is needed to set the Question Type (QTYPE) parameter to’comprehension." For 
example, 

IF GOAL = comprehension [222] 

THEN QTYPE = comprehension 

Next step: Select type of performance (Subgoal 2.3). 
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Subgoal 2.U: ApplicatioD 


Function: This goal is fued to ask an application question. 

Reasons for Goal Selection: 

1) If the (nevious question was an application question, and the content is the same. 
Example rule: 

IF GOAL s select question type [22] 

QTYPEPREV = application 
QCONTEST ^ same 
THEN GOAL = application [223] 

2) If the previous question was a comprehension question, and the content is new. 
Example rule: 

IF GOAL - select question type [22] 

QTYPEPREV = comprehension 
QCONTENT = new 
THEN GOAL s application [223] 

Outcome: A decision to ask an application questitm. 

Execution: Another rule is needed to set the QTYPE parameter to "application.* Fbr example, 

IF GOAL - application [223] 

THEN QTYPE = triplication 

Next step: Select type of performance (Subgoal 23). 
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Subgoal 2J.4: Analysis 


Function: This goal is fired to ask an analysis question. 

Reasons for Goal Selection: 

1) If the previous question was an analysis question, and the content is the same. 
Example rule: 

IF GOAL = select question type [22] 

QnPEPREV = analysis 
QCONTENT = same 
THEN GOAL-analysis [22.4] 

2) If the previous question was an application question, and the content is new. 
Example rule: 

IF GOAL = select question type [22] 

QTYPEPREV = application 
QCONTENT = new 
THEN GOAL = analysis [22.4] 

Outcome: A decision to ask an analysis question. 

Execution: Another rule is needed to set the QTYPE parameter to "analysis.” For example, 

IF GOAL - analysis [22.4] 

THEN QTYPE = analysis 

Next step: Select type of performance (Subgoal 2.3). 
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Subgoal 2^: Synthesis 

Function: This goal is fired to ask a synthesis question. 

Reasons for Goal Selection: 

1) If the previous question was a synthesis question, and the content is the same. 
Example rule; 

IF GOAL s se/ecr question type [22J 
QflYPEPREV = synthesis 
QCONTENT ^ same 
THEN GOAL ^ synthesis (225J 

2) If the previous question was an analysis question, and the content is new. 

Example rule: 

IF GOAL s select question type [22J 

QJYPEPREV e analysis 
QCONTENT = new 
THEN GOAL = synthesis [22J] 

Outcome: A decision to ask a synthesis question. 

Execution: Another rule is needed to set the QTYPE parameter to "synthesis.'' For example. 

IF GOAL = synthesis 1225] 

THEN QJYPE s synthesis 

Next step: Select type of performance (Subgoal 23). 
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Siibgoall^^ Evaluadon 


Function: This goal is fired to ask an evaluation question. 

Reasons for Goal Selection: 

1) If the previous question was an evaluation question, and the content is the same. 
E.'uunp** nile: 

IF GOAL s select question type 122] 

QTYPEPREV s evaluation 
QCONTENT ^ same 
THEN GOAL-evaluation [22.6] 

2) If the previous question was a synthesis question, and the content is new. 

Example rule: 

IF GOAL = select question type ]22J 
QTYPEPREV = synthesis 
QCONTENT^new 
THEN GOAL - evaluation [22.6] 

Outcome: A decision to ask an evaluation question. 

Execution: Another rule is needed to set the QTYPE parameter to "evaluation.” For example. 

IF GOAL = evaluation [22.6] 

THEN QTYPE » evaluation 

Next step: Select type of performance (Subgoal 2.3). 
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Subgoal 2J: Select type of perfonnance 

Function: This goal is fifed to select the t)rpe of performance to iequiie-«.g., peifarmanoe-based. text-based, or 
teflection/'articulatioo. 

Reasons for Goal Selection: 

1) If the question type has been selected. 

IF OflYPE * 0 

THEN GOAL = select type of performance [231 

Outcome: A decision to select the type of perfomiaoce lequiied of the student 

Next stq>: Selea one of the following • performance-based qtiestion4>roblem (Subgoal Z3.1), text-based 
questiof^proUeni {232), or reflection and articulation (2 J J). 
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Subgoal 23.1: Performance based question or problem 
Function: This goal is fired to select a perfonnance-based question or problem (e.g., a ^illation). 

Reasons for Goal SelecticMi: 

1) If the (Hevious problem was performance-based, and the content is the same. 

Example rule: 

IF GOAL - Select type of performance [23] 

QCONTENT same 
PTYPEtREV = simulation 

THEN GOAL - performance based question or problem [23.1 ] 

2) If the teaching method used was modeling, and the student is making nonnal progress through the Uq)ic. 
Example rule: 

IF GOAL = select type of performance [231 
METHOD = model 

TOPICPERF(topic x) = normal progression 
THEN GOAL = performance-based question or problem [2.3.1 J 

3) If the student lacks motivation, but has been performing well on the topic. 

Example rule: 

IF GOAL = select type of performance [23] 

MOTIVLEARN < criterion 
LOCPERF > criterion 

THEN GOAL = performance-based question or problem [2.3.1 ] 

Outcome: A decision to use a performance-based question or problem to assess the student's understanding of the 
topic. 

Execution: Another rule is needed to set the Performance Type (PTYPE) parameter to "performance." For 
example. 

IF GOAL = performance-based question or problem [2.3.1 j 

THEN PTYPE = performance 

Next step: Present the problem (Subgoal 2.4). 
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Subgoall3^: Text-based questioo or problem 
Function: This goal is fued to present a text-based question or problem (e.g.. multiple choice). 
Reasons for Goal Selection: 

1) If the previous question was text-based, and if the content is the same. 

Example rule: 

IF GOAL - select type of performance [23] 

QCONTENT ^ same 

PTYPEPREV s text-based question or problem 
THEN GOAL - text-based question or problem [232] 

2) If the student is just beginning to learn a topic. 

Example rule: 

IF GOAL = select type of performance [23] 

TOPICPROGRESS = early 

THEN GOAL s text-based question or problem [232] 

3) If the instructional presentation method involved telling. 

Example rule: 

IF GOAL - select type of performance [23] 

METHOD «tell 

THEN GOAL - text-based question or problem [232] 

Outcome: A decision to ask a text-based question or problem. 

Execution: An additional rule is needed to set the PTYPE parameter to "text-based.* Fc' example, 

IF GOAL - text-based question or problem [232] 

THEN PTYPE = text-based question or problem 

Next step: Present the problem (Subgoal 2.4). 
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Subgoal 2JJ: Reflcctun/Articulatioo 

Function: This goal is fired to require the student to reflect upon or aiticultfe cognitive processes (e.g., 'what have 
you learned today?) 

Reasons for Goal Selection: 

1) If the instructional period is almost over (e.g., the end of class). 

Example rule: 

IF GOAL - select type of performance PJJ 
TIMELEFT < criterion 
THEN GOAL - reflectionlarticulation [23 

2) If the student is nearly finished with the current tt^ic. 

Example rule: 

IF GOAL = select type of performance pJ] 

TOPICPROGRESS = nearly complete 
THEN GOAL - reflectionJarticuUaion [233] 

Outcome: A decision to ask the student to reflect on/articulate thoughts about the lesson 

Execution: An additional rule is needed to set the PTYre parameter to "reflection.* For example. 

IF GOAL - reflection [233] 

THEN PTYPE = r^ection 

Next step: Present the problem (Subgoal 2.4). 
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Subgoal 2.4: Present problem 


Function: This goal is fired to present a questioo or problem to the student, based on the parameters established in 
the earlier goals. The question or problem may either require a single reqxmse or a series of responses (e.g.. an 
instructional game or simulation). 


Reasotk' for Goal Selection: 


1) Performance type has already been selected. 

Example rule: 

IF PTYPE^ 0 

THEN GOAL a present problem [2.4] 

Outcome: A decision to presem the question or problem. 

Execution: Some mechanism for using these parameters to formulate a questioo or problem tteeds to be developed. 
For example, 

IF GOAL a present problem [2.4] 

THEN present problem within parameters specified by QCONTENT, QTYPE, and PTYPE. 

Next step: Diagnose student reqxxise (Goal 3). 
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Rules for Diugnosis 
GOAL 3: Diagnose student response 

Function: This goal diagnoses the student’s responses to the question^roblem from Goal 2. 

Reasons for Goal Selection: 

1) Problem has been {Hcsented to student 
Example rule: 

IF problem has been presented 
THEN GOAL * diagnose student response (31 

Outcome: A decision to diagnose the student's response. 

Next step: Collect student information (Subgoal 3.1). 
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Subgoal 3.1: Collect student information 
Function: This goal collects student le^ponse data. 

Reasons for Goal Selection: 

1) A decision to diagnose the student's response has been made. 

Example rule: 

IF GOAL = diagnose student response [3] 

THEN GOAL = collect student information [3.1] 

Outcome: A decision to collecc student information. 

Execution: Some mechanism is needed for accumulating and storing student data. For example. 

IF GOAL s collect student information [3.1] 

THEN collect data such as, ANS, TRIES, TIMEWAITING, and other performance data as needed. 
Next step: Provide feedback (Goal 4). 
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Rules for Providiog Feedback 
Goal 4: Provide iadividiialized feedback. 

Function: This goal provides individualized feedback at appropriate junctures, based on factors such as how long 
the student has taken to respond and how correct the answers have been. Feed b ac k is defined broadly to uiclude 
not intemqKing the student as a form of feedback. 

Reasons for Goal Selection: 

1) If diagnosis information has been collected. 

Example rule: 

IF (diagnosis information has been collected) 

THEN GOAL = provide individualized feedback [4] 

Outcome: A decision to provide some type of feedback. 

Next step: Either let the stuefent keep working on the problem (Subgoal 4.1) or interrupt the student (Subgoal 4.2). 
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Sttbgoal4.1: Do aot interrupt student 

Function: This goal is fued to decide not to iniemqx the student for the time being. 

Reasons for Goal Selection: 

1) If the student is still working on the problem, is performing adequately, but has still not mastered the 
u^ic. 

Ejcample rule: 

IF GOAL « provide individualuedfeedback [4] 

PERFSTATE > still working 
LOCPERF > criterion 
TOPICPERF (topic x) < mastery 
THEN GOAL do not interrupt student 14JJ 

Outcome: A decision not to iniemipt the student 

Execution: A rule is needed to continue diagnosis. For example, 

IF GOAL B do not interrupt student 14J] 

THEN GOAL - diagnose student response [3] 

Next step: Continue diagnosis (Goal 3). 
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Sal>goal4J: Interrupt the student 


Function: This goal is fired to intemipt the student 
Reasons for Goal Selectkm: 

1) The student has demonstrated mastery of the u^ic. 
Example rule: 

IF GOAL s provide individualized feedback 14] 

LOCPERF > criterion 
TOPICPERF (topic x) « mastery 
THEN GOAL = interrupt the student [42] 

2) The student is not performing well. 

Example rule: 

IF GOAL s provide individualized feedback [4] 

TRIES > criterion 
LOCPERF < criterion 
THEN GOAL = interrupt the student [42] 

3) The student is taking too long to respond to the question. 
Example rule: 

IF GOAL - provide individualized feedback [4] 

TIMEWATTING > criterion 
THEN GOAL a interrupt the student [42] 

Outcome: A decision to interrupt the student 

Next step: Select content of feedback (4.2.1) and present feedback ( 


150 




Subgoal 4 J.1: Select content of feedback 


Function: This goal is fund to select the components of the feedback (e.g., whether the response was conect or 
incorrect, what the coirect answer is, a short explanatimi, etc.)* That is, the feedback may include any or all of 
these components, depending on which rules fire. 

Reasons for Goal Selection: 

1) A decision to intemipt the student has been made. 

Example rule: 

IF GOAL - interrupt the student [42] 

THEN GOAL = select content of feedback [42.1 ] 

Outcome: A decision to selea the content of feedback. 

Next step: G}nsider including each of the following components in the feedback: information about 
conecmess^ncoirectness of response (Subgoal 4.2.1.1), the correct answer (4.2.1.2). a short explanation (4.2.1.3), 
praiseAeinforcement (4.2.1.4). performance data (4.2.1.5), and prompting self-review (4.2.1.6). 
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Subgoal Provide information about correctness or incorrectness of response 

Function: This goal is fired to provide information about the correctness or incorrectness of the student's respo n se. 

Reasons for Goal Selection: 

1) If goal is to select content of feedback. 

Example rule: 

IF GOAL = select content of feedback [42.1] 

THEN GOAL = provide information about correctness or incorrectness of response [42.1.1] 

Outcome: A decision to provide information about onrectness <»' inctmeciness of re^XMise. 

Execution: Another rule is needed to set Correct/Inconect Information (CIINFO) parameter to 'yes." For 
example, 

IF GOAL = provide information about correctness or incorrectness of response [42.1JJ 
THEN CIINFO^ yes 

Next step: Consider whether or not to provide the correct answer (Subgoal 4.2.1.2). 





Subgoal 4JLU: Provide correct answer 

Function: This goal is fired to provide the conect answer. 

Reasons for Goal Selection: 

1) If answer is incorrect or partially incotrecL 
Example rule: 

IF GOAL s select content of feedback [42.1 J 
ASS s incorrect OR partially correct 
TUBS GOAL = provide correct answer (42.12] 

Outcome: A decision to provide correct answer information. 

Execution: Another rule is needed to set the conect answer (CA) parameter to "yes.* For example. 

IF GOAL s provide correct answer [42.12] 

THEN CA«yes 

Next step: Consider whether or not to provide a explanation (Subgoal 4.2.1.3). 
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Subgoal 4.XU: Provide a short explanation 

Function: This goal is fired to {xovide a short explanation. 

Reasons for Goal Selection: 

1) If answer is incorrect or partially incorrect. 

Example rule: 

IF GOAL = select content of feedback [42.!] 

ANS = incorrect OR partially correct 
THEN GOAL s provide a short explanation [42.13] 

Outcome: A decision to provide a short explanation. 

Exec Jtion: Another rule is needed to set the Provide Explanation (PE) parameter to "yes.” For example, 

IF GOAL s provide a short explanation [42.13] 

THEN PE ^ yes 

Next step: Consider whether or not to provide praiseAeinforcement (Subgoal 4.2.1.4). 
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Subgoal 4JL1.4: Provide praise/reinforcement 
Function: This goal is fired U) provide praiseAeinfoicenienL 
Reasons for Goal Selection: 

1) If the answa was conect, and the problem was difficulL 
Example rule: 

IF GOAL = select content of feedback 142.1] 

ANS = correct 
PERFDIFF = difficult 

THEN GOAL - provide praiselrei/forcement ]42.1.4] 

2) If the answer was correct and the problem was of intomediate difficulty, but the student is a low aptitude 
student 

Example rule: 

IF GOAL - select content of feedback 142.1] 

ANS * correct 
PERFDIFF - intermediate 
AFT < criterion 

THEN GOAL = provide praise/reitforcement (42.1.4] 

Outcome: A decision to provide praiseAeinforcement 

Execution: Another rule is needed to set the Praise/Reinforcement (PR) parameter to *yes.* For example. 

W GOAL - provide praise/reinforcement (42.1.4] 

THEN PR = yes 

Next step: Consider whether or not to provide performance data (Subgoal 4.2.1 
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Sabgoal 4^1 JS: Provide perforiDaBce data 

Function: This goal is fired to provide peiftmnance (e.g.. scores, summary data). 

Reasons for Goal Selection: 

1) If a certain number of questions have been asked. 

Example rule: 

iF GOAL = select content of feedback [42.1 J 
NUMQUEST > criterion 

THEN GOAL - provide performance data [42.1 S[ 

Outcome: A decision to provide performance data. 

Execution: Another rule is needed to set the Performance Data (PD) parameter to *yes.* For example, 

IF GOAL = provide performance data [42 JS] 

THEN PD yes 

Next step: Consider whether or not to prompt the student to self-review (Subgoal 4.2.1.6). 
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Sabgoal htMapt adf*rcview 


Functioii: This goal is fired to piompt the ttudem to review his or ber response. 

Reasons for Goal Selection: 

1) If the student has been performing satisfactorily, but then answen a proUemincanectly. 
Example rule: 

IF GOAL B select content of feedback 142.11 

ANS B incorrect OR partially correct 
TOPICPERF(topic x) > normal progression 
LOCPERF > criterion 

THEN GOAL s prompt self-review 142.1.6J 
Outcome: A decision to prompt self-review. 

Execution: Another rule is needed to set the SeifReview (SR) parameter to’yes." For example, 

IF GOAL - prompt self-review [42.1.61 
THEN SR* yes 


Next step: Present feedback according to the parameters identified (Subgoal 4.2.2). 






Subgoal 4JZJ: Present respoase 


Function: This goal is fired lo present feedback according to the parameters established in Subgoal 4.2.1. 

Reasons for Goal Selection: 

1) If feedback para m eters have been selected. 

Example rule: 

IF (feedback parameters have been selected} 

THEN GOAL - present response [422] 

Outcome: A decision to {sesent feedback. 

Execution: Some mechanism is needed to provide feedback based on the parameters identified. For example, 
IF GOAL - present response [422] 

THEN present response within parameters specified by CIINFO, CA, PE, PR. PD, and SR. 

Next step: Reset variables for the next circuit through the model (Subgoal 42.3). 
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Subgoal 4.2J: Reset variables for next goal 

Function: This goal is fired to reset the variables for tbe next cycle through the modcL Variables are reset for 
three conditions-to |»escnt more informatitMi on the same topic, to elicit more perfonnance on the same topic, or 
to iHCsent more information on a new topic. 

Reasons for Goal Selection: 

1) If feedback has been presented. 

Example rule: 

IF GOAL - present response [422] 
response has been presented 
THEN GOAL = reset variables for next goal 1423] 

Outcome: A decision to reset variables for the next circuit through the model 

Next step: Reset variables to either return to Goal 1 within the same topic (Subgoal 42.3.1). return to Goal 2 
within the same topic (4.2.3.2). or return to Goal 1 with a change in topics (4.2.3.3). 


I 
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Subgoal 4JJ.1: Reset variables to return to Goal 1 

Function: This goal is fired to reset variables for another circuit through the model within the same tope • ei th e r 
for clarification or to present another aspect of the topic. 

Reasons for Goal Selection: 

1) If topic presentation is not yet complete, but the student is making s atis f acto r y progress. 

Example rule: 

IF GOAL - reset variables for next goal 1423J 

TOPICPROGRESS ~ early OR midway OR nearly complete 
LOCPERf > criterion 

THEN GOAL = reset variables to return to Goat 1 [423.1} 

Outcome: A decision to reset variables to return to Goal 1 within the same topic. 

Execution: A rule is needed to reset the variables; Fw example. 

IF GOAL = reset variables to return to Goal 1 [423.1] 

THEN reset the following variables to their defaults: (list of variables] 

Next step: Provide situated instruction (Goal 1). 
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Subgoal 4232: Reset variables to return to Goal 2 

Function; This goal is fired to elicit perfonnance again on the same topic - cither to give the student more practice, 
or because the student did not perform well previousty. 

Reasons for Goal Selection; 

1) If topic presentation is not complete, and the student needs m(»e practice. 

Example rule; 

/F GOAL s reset variables for next goal 1423] 

TOPICPROGRESS = early OR midway OR nearly complete 
LOCPERF < criterion 

THEN GOAL = reset variables to return to Goal 2 [4232] 

Outcome; A decision to reset variables to return to Goal 2 within the same topic. 

Execution; A rule is needed to reset the variables. For example. 

IF GOAL = reset variables to return to Goal 2 [4232] 

THEN reset the following variables to their defaults: (list of variables) 

Next step; Elicit perfonnance (Goal 2). 
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Subgoal 4J233: Reset variables to start new topic 

Function: This goal is fired to advance to the next topic within the course/cuiriculum and reset variables for 
another complete cycle through the model 

Reasons for Goal Selection: 

1) If student has mastered the current topic. 

Example rule: 

IF GOAL - reset variables for next goal 1423J 
TOPICPROGRESS = complete 
LOCPERF > criterion 
TOPICPERF(topic x) = mastery 
THEN GOAL - reset variables to start new topic [4233- 

Outcome: A decision to reset variables and provide situated instruction on a new topic. 

Execution: A rule is needed to reset the variables. 

IF GOAL ~ reset variables to start new topic [4233] 

THEN reset the following variables to their defaults: (list of variables) 

CURRTOPIC = topic x + 1 

Next step: Provide situated instruction (Goal 1). 
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